Book a Covid Vaccine live assessment
Service Standard assessment report Book a Covid Vaccine 24/11/2022
Service Standard assessment report
Book a Covid Vaccine
| From: | Central Digital & Data Office (CDDO) |
| Assessment date: | 24/11/2022 |
| Stage: | Live |
| Result: | Met |
| Service provider: | NHS Digital |
Previous assessment reports
This service has previously been through the Covid-19 rapid peer review process. These peer review reports are not published.
Service description
This service aims to solve the problem of…
- As a member of the public, I need to book an appointment so that I can receive a coronavirus vaccination
- As a member of the public, I need to book an appointment so that I can receive a flu vaccination
- As a member of the public, I need to book an appointment so that I can tell the NHS about a coronavirus vaccination received abroad
- As a member of the public, I need to cancel an appointment so that the NHS know that I will not be attending
- As a member of the public, I need to reschedule an appointment so that I can receive a vaccination at a different time
- As an arrival steward, I need to find an appointment so that I can check an individual in for their coronavirus vaccination appointment
Service users
This service is for…
- Members of the public
- NHS staff managing vaccination centres & appointments
- Staff checking in patients on arrival for appointments
1. Understand users and their needs
Decision
The service met point 1 of the Standard.
What the team has done well
The panel was impressed that:
- despite the time constraints, the service team managed to conduct quality user research to ensure all major pain points raised by end-users were quickly documented, analysed, and addressed accordingly
- the team have a deep understanding of current pain points for users in terms of finding a location, time that suits, and access needs
- the team has used the research insights gathered to improve the way users can cancel and manage their bookings
- the team has a good understanding of their internal users’ needs, and have a forward plan to improve searching at check in, all checks on a single page and reducing the number of clicks
- the team has a robust research plan in place for their key priorities that will continuously improve the Covid Vaccine and Flu booking services for both Citizens and internal staff
- the team collaborated well with other NHS user researchers and leveraged their key learnings and best practices to help improve their booking services
- the team demonstrated a deep understanding of their users, and presented well defined personas reflecting the user needs of general users as well as potential proxy users
What the team needs to explore
Before their next assessment, the team needs to:
- continue researching causes of the high drop-out/incompletion rates, and test their hypothesis on what design or content changes can help reduce this
- test the prototype with proxy users, to better understand how best to meet their user needs when using the NHS Vaccine and Flu booking services on behalf of someone else
- ways to test and iterate the group bookings/cancellations features quickly, as it will dramatically improve the efficiency of the service for NBS booking staff and internal staff
- explore using alternative research methods, for example, a co-design workshops with pharmacies and NBS booking staff would yield great insights and increase engagement
- it may be an idea to gather some user research on how far people are prepared to travel as the radius is currently 60 miles from postcode which may account for the 65% drop off rate when site location and appointments are presented. Maybe consider reducing 60 miles or if unsuitable location for travel is presented loop back immediately to reduce miles from postcode. It would then allow the analytics to really focus on why people drop off
- start thinking about how the needs for the service may change and moving towards a single Book A Vaccine service for multiple types of vaccine and other types of appointments – will it need to be more of an “Amazon” shopping experience for multiple bookings managed in a single service
2. Solve a whole problem for users
Decision
The service met point 2 of the Standard.
What the team has done well
The panel was impressed that:
- the team is minimising the number of times users have to provide the same information, for example by pulling information on a user’s vaccination history before allowing them to make booking, based on the user’s NHS number or personal details
- the sub-services join up into a cohesive whole: for example, Q-Flow for managing booking and vaccine availability data; the COVID and flu booking apps for citizens to draw on that data to book an appointment; and the check-in app to ensure the right people have arrived for each appointment
- the team is now looking to make sure the start page still meets user needs as we move from a pandemic to an endemic situation with COVID
What the team needs to explore
Before their next assessment, the team needs to:
- keep an eye on the scope of the overall service (or suite of services) when expanding it to include booking of other types of appointment, for example scans; scope creep could lead to a bloated NBS where the user experience of booking individual appointment types is negatively impacted
- monitor the need for additional call handler content – your slide deck says you rejected a “119 request to add call handler facing content on a citizen facing service” but are you sure there isn’t a need for such content in a call handler-facing area?
- how many people actually know their NHS number? If surname and date of birth always works if NHS number has low volumes as an option then perhaps don’t collect at all
- if the future service design is a single service approach where you can book multiple vaccines or other types of appointments – how you transition to that needs to minimise impact on your external users. Decisions on how need to be user lead and not delivery or technology lead
3. Provide a joined-up experience across all channels
Decision
The service met point 3 of the Standard.
What the team has done well
The panel was impressed that:
- the organisation of the team, and your overview of how you work, demonstrates that the service team is empowered to find the best way of solving problems.
- the team has designers and user researchers are working with front line operations staff to understand their user needs
- the team has developed a training tool so that call handlers (front line staff) can see how the service works and understand the users’ journey
- the sub-services of NBS join up to provide a seamless experience for the end user, for example, from making a booking in one of the booking apps to receiving a paper reminder to being check in at the site by a staff member using the check-in app
What the team needs to explore
Before their next assessment, the team needs to:
- for those who use the 119 service are we collecting feedback on why they are not using the digital service although low at 8% - could this provide further insight into how to improve the digital service or an accepted level of need for Assisted Digital
- consider how best to implement consistent and ongoing monitoring of the quality and timeliness of data input from individual sites and PCNs (you told us some surveys sometimes reveal inconsistencies, and that an investigation is in the pipeline)
4. Make the service simple to use
Decision
The service met point 4 of the Standard.
What the team has done well
The panel was impressed that:
- all content is designed by a dedicated content designer
- the user journeys for the public-facing booking services are genuinely simple to use and complete
- the recent improvements to the check-in app (‘pen and paper format ‘listing the day’s appointments instead of having search for them) look like a major improvement that will save NHS staff a lot of time and make the experience much smoother for patients
- there has been testing with a range of actual users (and 85mn booking so far!)
- the team has adhered to the NHS design system and content style guide throughout the service (with the exception of the Q-Flow sub-service)
- there is a robust and well-defined user-centred design process
What the team needs to explore
Before their next assessment, the team needs to:
- ensure that the sole content designer does not become overstretched as the service expands
- ensure that the addition of new products to the wider booking service does not make it harder for users to find the specific service they need
- where there are no appointments available nearby, consider including a message to reiterate that this is because sites nearby have no vaccine or availability, to reduce drop-offs
5. Make sure everyone can use the service
Decision
The service met point 5 of the Standard.
What the team has done well
The panel was impressed that:
- the team has fixed issues raised in their accessibility audit and (with the exception of Q-Flow) the services are now WCAG compliant
- the team has tested the service with people who have a range of access needs, including dyslexia, learning disabilities, ESOL, and parents with young children.
- the online booking service takes into account users’ self-declared access needs when selecting a venue for their vaccination
What the team needs to explore
Before their next assessment, the team needs to:
- review Q-Flow to ensure there are no accessibility issues for users, even if it’s not possible to achieve strict WCAG compliance
- continue to address the pain points for people with lower literacy: are there alternative or additional words you can use alongside the necessary medical terms?
- keep looking for ways to conduct more user research with (very busy) staff members
6. Have a multidisciplinary team
Decision
The service met point 6 of the Standard.
What the team has done well
The panel was impressed that:
- working in an agile and iterative way with a team structure which works for the scale of the services being worked on. The right skills and capability are in place at the right level
- the team feel motivated and empowered and passionate about the service they are continuing to improve
- the team understand what is on the backlog or in the plan as they move forward and understand the constraints in which they are operating
- there have been 3 different teams since inception due to the need to build this service as an emergency response to the pandemic but they have affected a good transition or handover which has minimised impact on pace of delivery
What the team needs to explore
Before their next assessment, the team needs to:
- continue to understand the constraints/impacts and their risks of continually developing the service
- consider how resource and capability within the team may need to adapt or change with future service design as you look at a single service for vaccines and other types of appointments
7. Use agile ways of working
Decision
The service met point 7 of the Standard.
What the team has done well
The panel was impressed that:
- all the key ceremonies are being used by the teams and there is a cross team stand up weekly to bring everyone together
What the team needs to explore
Before their next assessment, the team needs to:
- think about how the way the teams currently work may need to change to deliver a future service design - if service adds on further vaccine types or other types of appointments will the current model of team per dual services work(?)
8. Iterate and improve frequently
Decision
The service met point 8 of the Standard.
What the team has done well
The panel was impressed that:
- the team are in a continual cycle of iterating and improving the service in a user centric way
What the team needs to explore
Before their next assessment, the team needs to:
- think about defining Done or Measures of Success for the current services to know when the service is as good as it can be particularly as you need to balance further effort in continuous improvement of current services V investment in future service design
9. Create a secure service which protects users’ privacy
Decision
The service met point 9 of the Standard.
What the team has done well
The panel was impressed that:
- the team has automatic detection and notification of vulnerabilities in third party libraries
- there is no access through production accounts as processes are fully automated.
- there is good resilience in the network technologies provided by “Akamai” Global service for cybersecurity particularly for DDoS protectors.
- Akamai provides the capability to shutter the service during problematic situations.
- the service is deployed across 2 regions for maximum resilience. Regions -UK-West and UK-South
- there is a Disaster Recovery programme
What the team needs to explore
Before their next assessment, the team needs to:
- while there has been a penetration testing exercise; for clear clarity, there further should be orchestrated liaison with the National Cyber Security and other cryptography experts; for the ecosystem -the architecture team works closely with NHS Digital team provisioning a full cyber security test by an external CHECK approved organisation and continue to keep up to date with security-related best practice
- before the next assessment, the team needs to: For staff, ensure there is clear induction of skills, for building policies, working practices, guidelines and technology know-how, between contractors and permanent staff for new incumbents of the team for both Dev and BAU
- clearly define the DPO responsible to monitoring internal compliance across the whole ecosystem for inform and advise on data protection obligations, providing advice regarding Data Protection Impact Assessments (DPIAs) and act as a contact point for data subjects and the Information Commissioner’s Office (ICO) for the Ecosystem as a unit
10. Define what success looks like and publish performance data
Decision
The service met point 10 of the Standard.
What the team has done well
The panel was impressed that:
- the team collect extensive service performance and reliability data and share this within the team and across the organisation
- the team have ready access to data on digital take-up and completion rates. Cost-per-vaccination is well understood for the end-to-end service
- the team have acted on the data (in particular the high drop-off rate in the ‘choose a vaccination centre’ step) and made improvements to the service to reduce this
- the team have given thought to success measures and meaningful performance indicators for the service, spanning ‘improve health outcomes’, ‘improve people’s experience of health, care and wellbeing’ and ‘reduce pressure on the front line”
What the team needs to explore
Before their next assessment, the team needs to:
- define Measures of Success for current service and understand when you are Done with Continuous Improvement particularly with switch of effort to future design
- consider how to surface key service performance data externally to help improve transparency.
- use the data to ‘tell a story’ to provide interpretation and insight alongside the data and better illustrate the impact of incremental changes to the service
- continue to understand and monitor the cost and cost-per-transaction of the service as it is extended to other types of bookings, including measuring the contribution of the Booking Service to driving down overall cost
11. Choose the right tools and technology
Decision
The service met point 11 of the Standard.
What the team has done well
The panel was impressed that:
- the team considered a variety of architectures that could potentially address user and organisational needs, including a central database and federated communication model
- followed the GDS recommendation of Cloud first with Web Apps;
- choose for functionality required, highest level of resilience and scalability
- has Api Management, Azure Search, Cosmos DB,
- are making use of Cloud services
- appreciated their choice for Microsoft tech and hence MS Dev framework due to Azure being easier in terms of integration
- the team viewed traffic and chose a service that reflected the characteristics hosted
- the team uses familiar knowledge gained from the interim COVID-19 Digital Staff Passport service to inform technological decisions
- the team has consulted with the Chief Technology Officer
- the team has chosen an appropriate architecture within a complex space
- there is adherence to the WCAG 2.1, using automated tools for compliance.
- the team has planned mechanisms for frequently and quickly deploying changes across NHS organisations
- always looking to improve and managing code through backlogs
- design authority gives technical approval to the service
- NIMS is used for acquiring immunisation history dataCode is technical and decoupled from third party dependencies, a common system within NHS Digital offering
What the team needs to explore
Before their next assessment, the team needs to:
- carry on reflecting on technical choices in response to feedback during private beta
- consider whether decisions based on available skill set at the time may be revisited, such as “DevOps chosen were familiar with NHS UK so selected toolset on skills available, they had already shown that they could provide a service for booking”
- multiple changes in devOps staffing along with a high proportion of contractors opens up a requirement to ensure there is complete hand-over for the understanding of technology, policies and guidelines to each new incumbent
12. Make new source code open
Decision
The service met point 12 of the Standard.
What the team needs to explore
Before their next assessment, the team needs to:
- make sure it’s clear which Open-Source Initiative licence applies to released source code
- include usage instructions with source code
- more regularly release source code, ideally as part of the deployment workflow
- explore releasing the source code alongside other open-source code of a larger organisation
13. Use and contribute to open standards, common components and patterns
Decision
The service met point 13 of the Standard.
What the team has done well
The panel was impressed that:
-
the team has chosen the W3C Verifiable Data Model
- the team will use standard authentication mechanisms such as OAuth 2.0 and standard cryptographic methods based on elliptic curve public and private keys
- the team will host the system on the Azure cloud provider using standard hosting patterns, with blue/green deployments without downtime.
- the team built using the NIMS NHS -system Common component along with the PDS standards and GOV.UK/GOV.notify; Using the NHS design system (based on the GDS design system);
- provisioned in Azure Devops
- NHS style tool kit; java script and CSS elements
What the team needs to explore
Before their next assessment, the team needs to:
- implement and reflect on the planned technological patterns
- the team needs to elaborate further of tools used defined as “known capabilities” and common components within the ecosystem
14. Operate a reliable service
Decision
The service met point 14 of the Standard.
What the team has done well
The panel was impressed that:
- the team are operating a highly available and reliable service, with an excellent understanding of users’ needs to access the service at all hours, for example, due to the high proportion of users with caring responsibilities
- the team will have both testing and monitoring in place
- the service is an NHS Digital Platinum level service with 99.97% Uptime.
- 24/7 Support provided by NHS Digital Service Team
- team will have a pipeline in place to deploy both frequently and quickly
- the team understands which components must be available 24/7 and prioritises accordingly
- has Load balancing; Azure front door; DR testing: in UK region
- “3 Amigos approach” to design -pillars of technology, testing, product or user value
- protection by Azure front door for projection from the outcome of network attack
- uptime
- monitoring: Swop slots (Azure) supporting zero downtime deployments and enabling fast rollbacks; ITOC Information technology operation centre -platform for monitoring for alerts and logging
- CI/CD pipeline: Work with continuous integration and test automation to ensure early feedback; All tasks are repeatable and controlled and audited.
- AWS’s Terraform for Continuous development ; Deployment automatically through pipelines provisioned through Azure dev 0ps ;
- followed the process of the “Three Amigos” of technology, testing and user value are considered in everything
- testing: Smoke tested to certain the quality of releases
- network technologies provided by “Akamai” Global service for network protection Azure /AWS
- second level of defence is traffic management during high loading, controlling loading and providing info to the user when access will be available
What the team needs to explore
Before their next assessment, the team needs to:
- as part of future service design or moving towards a single service for booking vaccines and other types of appointments ensure that performance and resilience is scaled in the right way for increased transaction volumes and impact of integration to other services
- provide justification for the choice of the procurement of suppliers
- resolve medium highlighted risks uncovered during penetration testing
- clear timescale required for mitigation of medium classified risks uncovered in penetration testing
- detailed collaboration for ownership of risks in the ecosystem
Updates to this page
Published 24 January 2024