Platform as a Service (PaaS) - Alpha Assessment
The report from the alpha assessment of the GDS’s Platform as a Service (PaaS) on 23 June 2016.
| Stage | Alpha |
| Result | Met |
| Service provider | Government Digital Service (GDS) |
The service met the Standard because it:
-
Identified the common need across government for centralised and automated hosting, deployment and testing of code so service teams can focus on development.
-
Assessed and tested multiple solutions during the alpha whilst keeping focus on their minimum viable product at this early stage.
About the service
Service Manager: Rory Hanratty
Digital Leader: Chris Ferguson
The service provides a central infrastructure platform and command line interface for ease of deployment.
Detail of the assessment
Lead Assessor: Scott Colfer
User research assessor: Angela Collins-Rees
Technical assessor: Al Davidson
Design assessor: Tim Paul
User needs
The service team were able to articulate key user needs and demonstrated an adequate amount of user research to meet the alpha standard. However, the team have not maintained regular user research throughout the whole alpha period. Without the intense effort during discovery and the start of alpha, and again in the last month, the team would not have met point 1.
The team now have a permanent user researcher in place and in the assessment presented a robust research plan for their private beta. The panel was reassured to see the team have a clear plan for addressing research questions that will inform their private beta.
The panel believe GDS must lead the way in respect to user research and set an example for other departments. Although the team understood the key user needs, the panel judge the service would have benefited from having a researcher on the team throughout the alpha. However, the panel also acknowledges that technical platforms, such as PaaS, may require a slightly different team configuration compared with public facing services.
Team
The team have not met the standard for point 3 because of the absence of a user researcher during some of the alpha. Platforms should be held to the same standard as public facing services and government departments should make professions like user research, content design, and design available to these teams throughout development.
The panel note that the PaaS team have worked hard to recruit a user researcher and recognise the need for additional skills in the team, such as a content designer.
Technology
The team made sensible technical choices during alpha, using open source products and trialling alternatives with users. Where they have needed to change code, patches have been submitted back to the public version of the product, ensuring code is published and free to use.
The team have taken the security of their potential clients very seriously, engaging with both in-house and external penetration testers, and a CESG architect. The panel notes that while there are still some data isolation issues to be addressed in beta, the team are sensibly restricting usage to services containing only public data until those issues are addressed.
The panel is confident the team are able to test their service fully and will be able to iterate with confidence during beta.
Design
The team demonstrated that most users are able to use the service first time, without assistance. However, further work to understand the needs of users that are less technically capable is needed to ensure the right type of support and documentation is provided. For example, consider designers that are familiar with editing front end prototypes but less familiar with setting up servers using the command line.
It’s crucial that the supporting technical documentation and command line messages are treated with the same level of attention as the rest of the service. The current documentation format is not consistent with the GOV.UK style. The team needs the support of an interaction designer and content designer for this. This is a requirement across the whole of the GaaP programme.
The team should revisit the service proposition (including the name) to check that it’s well understood across government, particularly in relation to other services like Crown Hosting.
Analytics
The team has been measuring KPIs during alpha and the panel is confident they will be iterated and improved based on what’s learned during beta.
Recommendations
User needs
The team needs a dedicated user researcher to ensure the platform is built to meet user needs and appropriate user testing takes place to confirm the needs are being met.
The panel was pleased to learn the team had identified some high level user needs from their discovery research and recommend the team presents them at the start of their next assessment.
The panel was concerned to learn that there had been a significant period when a user researcher was unavailable to support the team’s ongoing work. While the remaining members of the team were pragmatic in their approach to gathering user feedback, this raised concerns about the research approach, the user groups interviewed and the subsequent knowledge gaps.
The panel learnt that prior to the assessment, the team conducted a round of usability testing with 15 junior developers. The panel would like to see further evidence of a much deeper understanding of their users, not just those building the service but also those maintaining and supporting it. As part of their research and usability testing the team must include users and potential users with access needs and those who use assistive technology.
The service currently does not support .Net applications. This will prevent some areas of government using the PaaS. The panel recommend the team engage with these areas of government to understand their needs and incorporate that into their wider programme of work.
Team
GDS should ensure that it has the people and skills available for its teams to meet the service standard. A lack of the right people/skills was a significant blocker for the PaaS team, who were forced (at times) to do the best they could with the resources they had.
Technology
While the team have so far concentrated efforts on making the service highly-available, the panel recommend that they devote similar effort during beta to considering their support model, onboarding process, and other non-technical aspects of their service wrapper. We also recommend continued thinking around isolation of logging data, in relation to the kinds of client service they can support, and user-testing of the command-line interface itself as well as the documentation.
Analytics
The team should consider how to measure the amount of money saved as a result of service teams using the PaaS.
Recommendations
To pass the next assessment, the service team must:
-
Ensure that a user researcher is available to the team during beta and broaden their research to include users with access needs
-
Focus on the broader service design of the PaaS, with specific focus on the design of onboarding and documentation
-
Define the ‘market’ for PaaS and ensure that the technology and the service wrapper are resilient enough for growth
The service team should also:
-
Understand the needs of development team members less proficient in deployment (eg designers)
-
Put together a strategy for marketing and encouraging the adoption of PaaS
-
Plan how they will manage the expectations of early adopters who want to use PaaS but need to know what they can use and when they can use it