Driver examiner service alpha assessment
The report from the alpha assessment for DVSA's driver examiner service on 24 April 2018.
| From: | Central Digital and Data Office |
| Assessment date: | 24 April 2018 |
| Stage: | Alpha |
| Result: | Met |
| Service provider: | DfT DVSA |
The service met the Standard because:
- This is a capable team working in a collaborative and iterative fashion, led by an empowered service manager and with good relationships with those off team and governance in place.
- The team have built and continuously iterated a prototype based on ongoing user research to understand and meet user needs.
- The team have had a good technical approach during Alpha and the assessment team are satisfied with the technical choices and the work that has been carried out.
About the service
Description
A key factor in keeping Britain’s roads safe is ensuring that all drivers have the skills, knowledge and understanding to be a safe and responsible driver.
There are different types of test for different variations of vehicles including Cars, Buses, Lorries, Tracked vehicles, tractors and Motorcycles. In a separate service, candidates complete a test to demonstrate theory knowledge and hazard perception skills and once passed can book a practical test where an examiner will assess their driving ability and knowledge against specific criteria, this is where our (app) service begins.
The service currently accepts (among other channels) online test bookings, however for the examiner the process is completely manual and paper based.
The examiner’s daily schedule and associated candidate data is printed out at test centres daily where it is transposed manually to paper test forms to be used for examinations. During the test the faults are marked on the paper form, then once the test is complete the candidate gets the result and feedback is given orally. Once back in the office the examiner must write up test and candidate information manually. The paper forms are posted to Newcastle upon Tyne for scanning and population into the legacy system of record. This scanning process has a high failure rate and high manual intervention need. Following the scanning, DVLA issue a driving licence for successful licence acquisition candidates.
Service users
Prime: Candidates, DVSA Driving examiners
Secondary: Delegated examiners, Approved Driving Instructor examiners, Approved Driving Instructors Quality Assurance employees, Local Driving Test Centre Manager, Examiner Trainers, Deployment team member.
Detail
User needs
Using its domain knowledge and a well-balanced mix of research methods, the team is understanding who its users are and what they need. It is considering a broad range of people who the service impacts including driving examiners, candidates, driving instructors, training professionals, quality assurance professionals, and test centre managers.
The service users’ needs fall into a few categories. For one, examiners need to stay productive under time pressure; to avoid manual work like copying information from journals to DL25 forms, tallying scores, and rework like formulating and delivering feedback to candidates at the time of a test and later writing up that feedback, sometimes after it elapses from memory.
For another, they need to focus on evaluating the candidate’s performance and the test’s legality. Currently, examiners rely on their own timekeeping tools and handwritten lists to keep track of whether the conditions for a legal test are met. The paper from they currently use lacks fields to record these points and furthermore contains irrelevant fields that examiners consistently ignore.
In addition to examiners’ needs, the team described test centre managers’ need to process test results accurately and without delays, noting that they contend with backlogs of paper forms and sensitive scanning machines that sometimes calculate scores incorrectly.
The team is factoring in candidates’ needs to stay focused during the driving test into its decision to outfit the tablet with a privacy screen and silent stylus. It is also considering candidates’ need to receive feedback at a time when they are well-placed to process it, and in a format that helps them learn. Yet, the team need to do more research with candidates, to check that the new system is no more distracting than the current paper based versions, and to find out how and when they want to receive feedback about mistakes during the test.
Another category of need, to stay safe on the road, is an explicit goal of the DVSA, but also an implicit need of service users and of road users generally. The team must prioritise research to understand this need in the context of a driving test. Particularly in the period where examiners adjust to the new way of doing things, the team should anticipate that overcoming old habits will consume attention beyond examiners’ usual focus on the candidate’s performance and on the road.
The team must do research to evaluate how the digital service and its associated hardware, training, and support model are performing against the need to protect road users. It must take care to cohort users by their level of exposure to the service and make sure that examiners with the lowest levels of exposure and digital skill are able to record faults whilst maintaining awareness of the road.
The team is starting to research how examiners interact with the service as they log typical faults. It must now research how examiners get on with tasks that they do infrequently, tasks that are cognitively demanding like making and correcting mistakes, or that require examiners to resolve ambiguity.
To evaluate the prototype, the team is considering how useful or how easy examiners perceive the service to be. Because perceptions of safety and ease don’t guarantee ease and safety, the team must identify appropriate metrics and evaluate how easily and safely examiners can accomplish tasks whilst maintaining an awareness of the road. Beyond considering examiners’ stated training needs, the team must evaluate what kind of training and how much of it enables examiners to accomplish tasks whilst maintaining an awareness of the road.
The team has identified that 1 in 6 examiners feel that the app will be harder to use than the current paper based system. These are also the hardest to reach through user research participation. The team must find ways to gather feedback from these users and incorporate their needs into the service.
Team
The assessors are satisfied that the team is resourced for current phase and sufficiently evidenced that they are working in an agile way.
The service team clearly articulated the governance and the relationship that each of the boards and authorities have with the service.
The team must include content designers during the beta.
Currently user research is carried out by the designer, and these need to become separate roles in the beta. It is worthwhile thinking about hiring or contracting a specialist in iOS app UI design to speed up design iteration.
The team is currently made up of a mix of DVSA staff and contractors. There is a good approach in place for knowledge transfer and a plan to introduce more DVSA staff as the service progresses.
The service has a clear plan for the service being offline and what actions users will see and have to do at this point.
The team have an approach in place to drive digital take up. They would benefit from considering in more detail how they intend to address the needs of the users who have limited or no exposure to Tablets or PCs, and use this feedback to help iterate and improve the service.
Technology
The assessor team are generally satisfied about the technical choices and work that have been carried out by the service team.
The service team have provided convincing evidence that they have considered all aspects of the service manual in order to properly kick off the Beta phase.
Even though native apps are usually regarded as not suitable for a government service, the specific requirements of the Driver Examiner Service provide a convincing argument that a web service wouldn’t be suitable due to a lack of required functionality.
It’s laudable that despite having to build a proprietary apps, the service team used to build it using the cordova framework, which lets developers write apps as standard web applications, using HTML, CSS and JavaScript.
The assessors also welcome the publication of the app’s code as open source on github during alpha. It shows that the service team cares about openness and also confirms that the team’s developers use modern code writing practices.
Recommendations for Beta:
Because this is likely to come up again, the team should provide exhaustive evidence that designing a native app was the only way to respond to the needs of the service’s users or specific legal requirements. Refer to https://gds.blog.gov.uk/2013/03/12/were-not-appy-not-appy-at-all/
Given that the choice of hardware seems to be outside of the service team’s remit, it is important to have a feasible way to port the app to another device, or possibly make it a web app. The team should be able to demonstrate that they do.
The service team should know exactly what trust they should put in the services used (AWS lambdas, Airwatch). It’s important to know exactly how some key features work, such as over-the-air encryption of data between the device and the backend, in order to make sure the service is secure against attacks, and not rely on an external ITHC. Independent monitoring of the stack (not relying on AWS directly, such as using Pingdom), including message queue, is also strongly recommended to avoid possible loss of traffic due to patchy connectivity.
Now that the team has registered with the Performance Platform, establishing what data can be collected from the system and shared publicly should be done early as it will influence how the software is built. Popular examples include cost per transaction, or number of transactions processed.
More testing will be needed to make beta robust: regression testing in a production environment, as part of the continuous integration process, as well as testing for responsiveness of the app. For instance, UI lag due to high system load, low memory space or battery level, or the fact that the app uses a framework and not iOS’s native language. The team seem aware that it is something to monitor closely.
Building interfaces with outsourced legacy services, is often an impediment to iterating the API in an agile way. The service team should remain aware of possible delays or general constraints incurred from working with suppliers.
Design/content
The team has made a great start on designing the app for driving examiners, concentrating on the two hardest screens; the journal, and the in-test mode.
There is not currently a clear design system and UI style for the application. There will need to be considerable work on creating a robust usable style for the app - it was mentioned that the design was not trying to be like iOS, as many examiners will be new to the OS and tablets; however, DVSA is expecting them to use several other apps on the tablet, and so some OS standardisation would be helpful. The design should also be careful to make clear what are buttons and are not (if everything is rounded, everything looks like a button), placement and differentiation of primary actions and undo/back, and be careful that it is accessible (not differentiating just with colour). The team should also research carefully use of icons (as seen in the main navigation bar without legends) as they are often misunderstood.
The work that the team has done to iterate the in-test mode of the app is good and is based on research in real-life use. However, more radical design could be considered for this screen - the graphic and interaction design is not currently making use of standard visual differentiators - eg section designation, text size, colour - and there should be more research on the base interaction (long press, double select for serious and dangerous faults). In particular, making a change to an error has been thought about but is not as easy as on the current paper form. This is a very specialist screen, and very different from the rest of the app, so the team should not be concerned with keeping the design of this screen in line with the rest of the app. It is also important to iterate this heavily during the beta, as this screen needs to be fixed so examiners can learn to use it and the placement of buttons, so changes to it should only happen rarely when actually being used in public beta/live. The team should also find out if examiners need any extra functionality on this screen, such as changes to brightness or ability to start and stop the clock.
The current screen designs look good, but needs to be productionised. The team should take live cuts of data from examiner’s real journals to check that the graphic design will work on all screens, and that text overflows are handled appropriately.
The team should also check the quality of information brought into the system from other services eg candidate photo and signature to make sure they are usable.
The design must be checked for accessibility and that system-wide adjustments on the iPad (eg text size) will be reflected in the UI, and will not break the design. The app must also be designed to have some responsiveness - although it is being designed for one device, the device will be discontinued and newer devices may not have the same screen size or aspect ratio.
The team have done some speculative design around how to provide feedback to candidates after the test. This should be explored further and backed up with research. Further work should be done on how and when to contact candidates, including email, and how to provide the pass certificate or copy of the test.
Test centre managers are an important user group in the service, but currently no work has been done to improve their part of the the service. By beta there should a plan to modernise the service for managers and to make sure their work is has been improved by the service.
So far no content designer has worked on the app. This must happen in beta, being careful to rethink ingrained DVSA and DVLA terminology (eg DL25).
The rollout and private Beta plan for the devices and app has been well thought through. The team should find a way to distribute the app internally so that other groups of users (such as tech and app support teams) can experience the app and see how it works. The team must also be able to distribute the app to the assessment team for any future assessments.
Analytics
The team had a clear idea of the type of analytics and key performance indicators (KPI’s).
They have clearly thought about which KPI’s and metrics they need to measure on both the online and offline sections of the service and have looked at these beyond the mandatory four KPI’s.
The service would benefit from defining appropriate metrics to evaluate examiners ability to maintain awareness of the road whilst they interact with the service. These may include e.g. time it takes locate fields, time it takes to log faults, time it takes to correct mistakes, measured or reported stress levels.
Now that the team has registered with the Performance Platform, establishing what data can be collected from the system and shared publicly should be done early as it will influence how the software is built. Popular examples include cost per transaction, or number of transactions processed.
Recommendations
To pass the assessment, the service team must:
- Provide exhaustive evidence that designing a native app was the only way to respond to the needs of the service’s users or specific legal requirements
- Identify and provide metrics to be shared publicly on GOV.UK’s performance platform
- Hire a content designer and make sure the app is well worded and understood
- Have separate designers and user researchers in each development team
- Have a clear UI design system for the app based on usability and user research
- Make sure the application and service is accessible. Research and test the service with accessible users and user on the lower end of the digital scale. Make sure the application and service is accessible.
- Do research with candidates to ensure their experience of the service has been improved throughout
- Find ways to do research with examiners who feel the new system will be harder to use or have less experience with tablets and computers.
- Address, in its research and design work, the need for examiners, candidates and other road users have to stay safe on the road.
- Do research to evaluate examiners ability to notice and react to circumstances that arise on the road (including those that may arise when candidates are temporarily stopped) whilst they record manoeuvres, log faults, and correct mistakes.
- Define appropriate metrics to evaluate examiners ability to maintain awareness of the road whilst they interact with the service. (These may include e.g. time it takes locate fields, time it takes to log faults, time it takes to correct mistakes, measured or reported stress levels.)
- Do research to evaluate how much and what kind of training lets examiners get familiar enough with the service to maintain an awareness of the road whilst they record manoeuvres, log faults, and correct mistakes.
- Understand the characteristics that influence examiners’ ability to maintain awareness of the road whilst they interact with the service. (These may include motor or cognitive characteristics, level of experience with tablets, years of experience using the DL25A.) Research with users who across the range of these characteristics.
The service team should also:
- Work with a specialist iOS application designer to improve the UI and design system for the app.
- Iterate the in-test screen based on research under real examination conditions and explore ways to make the screen even more usable.
- Do research with candidates to ensure their experience of the service has been improved throughout.
- Find ways to do research with examiners who feel the new system will be harder to use or have less experience with tablets and computers.
- Address, in its research and design work, the implicit need that examiners, candidates and other road users have to stay safe.
- Do research to evaluate examiners ability to notice and react to circumstances that arise on the road (including those that may arise when candidates are temporarily stopped) whilst they record manoeuvres, log faults, and correct mistakes.
- Define appropriate metrics to evaluate examiners ability to maintain awareness of the road whilst they interact with the service. These may include e.g. time it takes locate fields, time it takes to log faults, time it takes to correct mistakes, measured or reported stress levels.
- Do research to evaluate how much and what kind of training lets examiners get familiar enough with the service to maintain an awareness of the road whilst they record manoeuvres, log faults, and correct mistakes.
- Understand the characteristics that influence examiners’ ability to maintain awareness of the road whilst they interact with the service. (These may include motor or cognitive characteristics, level of experience with tablets, years of experience using the DL25A.) Research users across a range of these characteristics.
- More testing will be needed to make beta robust: regression testing in a production environment, as part of the continuous integration process, as well as testing for responsiveness of the app. For instance, UI lag due to high system load, low memory space or battery level, or the fact that the app uses a framework and not iOS’s native language. The team seem aware that it is something to monitor closely.
Next Steps
You should follow the recommendations made in this report before arranging your next assessment.
Get advice and guidance
The team can get advice and guidance on the next stage of development by:
- searching the Government Service Design Manual
- asking the cross-government slack community
- contacting the Service Assessment team