Patient appointment booking service
The report from the alpha reassessment for NHS Digital's patient appointment booking service on 24 May 2017.
| From: | Central Digital and Data Office |
| Assessment date: | 24 May 2017 |
| Stage: | Alpha - reassessment |
| Result: | Met |
| Service provider: | NHS Digital |
The service met the Standard because:
- The service was assessed against the criteria for Alpha.
- The team had made good progress since their original assessment, and addressed concerns raised.
- The team demonstrated an excellent awareness of their users’ needs, and clearly have embedded user-centric design into their development process.
- Where the panel had concerns about the service, these related to criteria relevant to a Beta assessment, and arose only because the service is exceptionally already in the public domain.
About the service
The service allows patients to book a hospital appointment following referral by a GP.
Detail of the assessment
User needs
Since the last assessment the team have continued their previous good work in understanding users of the service. In addition to pop-up research with people in hospital waiting rooms, they have expanded their work to include in-home research with patients who have significant health conditions and it is clear that this research has proved central to the development of the service.
The team are aware of the need to conduct accessibility testing and this has been scheduled for June. In addition to this they were able to explain the alternative pathways and support that claimants can use to manage their appointments if the online service was not useable for them.
Although the letters inviting patients to manage their appointments online are not strictly part of this work, it is clear that they are being reviewed and research is a key part of that. The team emphasised that work across all parts of the service is linked not just at the service manager level but within individual disciplines like research and design.
Overall the importance to the product development team of understanding user needs was very apparent. They were able to give multiple examples of how research has changed the direction of development challenging prior assumptions. The product owner also explained how research plays a significant role in the prioritisation of features.
As the team moves into Beta they will have more opportunity to learn from the analytics they are recording and can use this to guide further qualitative work. It will also be an opportunity learn about broken journeys where patients have started online but then failed to complete a booking before moving to telephony. Follow up interviews can ascertain whether this is a result of availability (as suspected) or something else.
Team
The team demonstrated a very strong understanding of the users’ needs for their service, and were clearly committed to meeting them. They are working in an agile way, with some constraints resulting from legacy technology and working practices, and are making good progress resolving blockers as they find them.
The team is clearly well led, with highly capable and empowered Product and Service Managers. We were also particularly impressed at the extent to which thorough and well-considered user research was embedded in development practices.
Overall we felt that this was an impressive team, finding its feet and working collaboratively on a service they are clearly passionate about. They should be supported and encouraged to continue along their current path.
Technology
The team demonstrated a move towards faster release cycles and a strong intention to further increase the speed at which they can deploy changes to the patient application.
Whist the team has made some progress in opening their code (working to define governance within their department) they stated that they would not be likely to open source any code until they had taken steps to upgrade some outdated libraries. The stated reason for this was to prevent attackers being able to identify the software they were running and to exploit vulnerabilities in it.
This does not provide a strong justification for keeping the code closed for a number of reasons. Firstly for any client-side code it is trivial for an attacker to determine the version of the framework used - so closing adds no real prevention here. Secondly for server side code there should be existing mitigations in place to prevent attackers exploiting known vulnerabilities. Security through obscurity is not an effective defence against attack.
For the alpha the team stated that iterating on the existing authentication mechanism was out of scope. For the beta I would suggest the team work with the Citizen Identity Team at the NHS in order to determine if a more robust approach to identity assurance may be appropriate
Design
The team demonstrated they had an understanding of the wider user journey and had considered the referrals service fits into this context. They have mapped the existing journey to identify pain points and have also mapped out a future version of the service to identify areas to developer.
The panel felt there was a security risk with relying on passwords printed on paper, especially as only a small minority of users change their passwords.
The team described how they had undertaken pre-discovery work around the letters, including visiting elderly users at a Jewish community centre, and that they had been in contact with the GOV.UK Notify team to understand the services they offered. The team should continue to identify alternative authentication solutions.
The service has been renamed, and the verb “manage” was chosen as the service incorporates more actions than purely booking an appointment.
As the service is being used by large volumes of users, the team has access to detailed metrics about usage and have observed a positive increase in completion rates. However, the panel is concerned that the team will not be able to make iterations and try out new ideas whilst the service is ostenisibly in a public beta phase.
Analytics
The team demonstrated a strong understanding of the value of good quality analytics, and have been using the high level of usage (for an Alpha) to provide valuable insight for the service. They have iterated their toolset, from Splunk to Piwik, and have clearly placed significant emphasis on gathering quantitative data to lend clarity to the qualitative information gathered via User Research. For an Alpha, their approach is well-developed.
Recommendations
To pass the next assessment, the service team must:
- Have some code open sourced. The patient application Javascript code seems like a good candidate. As well as being good practice for public sector software development purposes, it is also good discipline and will help to ensure that code receives appropriate scrutiny.
- Have explored alternatives to the printed password authentication mechanism for patient log-in. The NHS ‘Citizen Auth’ team is working on providing a central authentication platform. If the team decide that this is not a good fit then compelling evidence should be given to support this choice.
- The service has effectively ‘skipped’ the ‘private Beta’ stage, and gone directly to ‘public Beta’. Whilst the panel understood why this has occurred, it is important that the team are able to demonstrate their ability to test alternative hypotheses with real users, to support continued iteration of the service design.
The service team should also:
- The panel strongly recommends that NHS digital arrange a Beta Workshop with the GDS assessment team no later than August 2017. Whilst the panel was content with the ‘Alpha’, there are a number of significant concerns relating to a future Beta assessment, and these should be worked through in detail, to ensure that they are appropriately addressed before a formal Beta assessment. As the service is already in the public domain (effectively beyond the Beta service assessment) it is vital that the significant concerns, particularly around service security, are properly resolved.
- For future service developments, we would also strong advise NHS Digital to avoid the lengthy delay between the original Alpha assessment and service reassessment.
Digital Service Standard points
| Point | Description | Result |
| 3 | Having a sustainable, multidisciplinary team in place | Met |
| 5 | Iterating and improving the service on a frequent basis | Met |
| 8 | Making code available as open source | Met |
| 12 | Creating a simple and intuitive service | Met |
| 15 | Using analytics tools to collect and act on performance data | Met |