Evaluation Registry beta assessment
Service Standard assessment report Evaluation Registry 08/07/2024
Service Standard assessment report
Evaluation Registry
| From: | Evaluation Task Force |
| Assessment date: | 08/07/2024 |
| Stage: | Beta |
| Result: | Red |
| Service provider: | Cabinet Office |
Previous assessment:
- Alpha assessment. May 2023. Met. Departmental assessment. Not published.
Service description
The Evaluation Registry will be the go to place to find and share evaluations across the Civil Service. Having easy access to evaluation insights will improve policy design and evaluation.
Civil servants are responsible for evaluating how a policy intervention performs, the Evaluation Task Force (ETF) in Cabinet Office is in place to ensure the evaluations are better used. During discovery the ETF identified that there needs to be a shared place where civil servants can share and search evaluations in a consistent way.
Service users
This service is for:
-
evidence creators (social researchers within Government)
-
evidence users (policy designers and decision-makers for example Treasury, within Government)
-
evidence compilers & system overseers (Evaluation Task Force and central evaluation teams in departments)
-
outside of government (National Audit Office, charities, campaigners, media)
Things the service team has done well:
-
technology: They have made improvements to the service framework that the previous team i.AI had developed. The architectural design and implementation of the service has been kept simple and effective.
-
the team has produced a Performance Framework document to determine the relevant KPIs needed to measure the success of the service. They demonstrated how any insights discovered will be fed back into the service team to help use data to drive improvement.
1. Understand users and their needs
Decision
The service was rated red for point 1 of the Standard.
During the assessment, we didn’t see evidence of:
-
conduct research with users who have additional needs. This should include but not be limited to people who use assistive technology, people who are neurodiverse and those for whom English is a second language. The sample should reflect what the team know about their users – both those who work in Government and members of the public.
-
the aim of public beta is to answer the question ‘does this work for everyone?’ Without understanding these needs the team cannot provide evidence this is the case.
-
conduct research with users outside Government who will be able to access evaluations once the team enters public beta. The panel were concerned that as the only new user group in the next phase, no work had been undertaken to understand their needs and whether the service would meet them.
-
the team were focussed on product functionality and had lost sight of the service outcomes. Having evaluations in one place adds some value in terms of improving how easy it is to find them. But significant value will only be attained if this leads to better decisions, less duplication, more transparency etc. Understanding the needs of external users is key to the service achieving these aims.
-
increase the cadence and breadth of user research. The team have conducted 18 sessions in an eight-month private beta, all focussed on prototype testing of new features. The team need to increase the breadth and depth of their User Research. This must include regular face to face work with people using the service in the real world.’
-
increase the rigour with which in-service feedback is being analysed and acted upon. The team gave the impression this was not being led by the User Researcher, and information was coming to them third hand from their policy colleagues.
-
be clearer on how they are prioritising what to iterate and the criteria used. It was difficult for the panel to understand which user needs had been met and which were still unmet.
-
ensure user needs and personas reflect the high-level needs of all users. They are heavily based on the low-level detail of the service.
2. Solve a whole problem for users
Decision
The service was rated amber for point 2 of the Standard.
During the assessment, we didn’t see evidence of:
-
do more research to understand if the public-facing service is easy or difficult to use for different groups of people.
-
revise the public viewer persona using real evidence from user research.
3. Provide a joined-up experience across all channels
Decision
The service was rated amber for point 3 of the Standard.
During the assessment, we didn’t see evidence of:
-
it’s not clear who is responsible for the content and guidance on GOV.UK. And how the team will know if the language and content is consistent throughout the service. If several departments are responsible for writing the content, a content style guide might be useful.
-
finish the service blueprint so you can identify more unhappy paths and improve the support model. It’s not clear how users, especially the general public, will get support.
-
it’s not clear how the team plans to collect feedback from the public-facing service and how that feedback will be assessed against goals and KPIs.
-
if the general public must submit a freedom of information request (if they can’t use the online service), the team must test this journey with users to make sure it works as part of the end to end service.
4. Make the service simple to use
Decision
The service was rated red for point 4 of the Standard.
During the assessment, we didn’t see evidence of:
-
there’s not enough evidence that the service is simple to use for everyone because of the gaps in user research (see point 1).
-
test all parts of the public facing service with real users, including support channels and unhappy paths. Show how you have created a consistent experience from start to finish.
-
make sure the public service uses the GOV.UK design system patterns and components (or designs from other government departments at least). If a new component was used, explain why and contribute it back to the design system so other people can reuse it. You need to explain why you have created bespoke components and patterns and prove they are accessible and usable.
-
attend a design crit with content designers and interaction designers currently working in government.
5. Make sure everyone can use the service
Decision
The service was rated red for point 5 of the Standard.
During the assessment, we didn’t see evidence of:
-
the team cannot guarantee that everyone can use the service because of the gaps in user research (see point 1).
-
rectify all issues identified in the DAC report and update the accessibility statement.
-
test the assistive digital route with disabled users or users who don’t want to use the online service for whatever reason.
6. Have a multidisciplinary team
Decision
The service was rated red for point 6 of the Standard.
During the assessment, we didn’t see evidence of:
-
any thought put into capability planning for the next immediate, medium or long term to support and develop this service.
-
continuity of a team to continue to develop the service once it moves into public beta
-
sustainable funding plan to continue to support and develop the service once it begins to be used by real users at scale.
7. Use agile ways of working
Decision
The service was rated amber for point 7 of the Standard.
During the assessment, we didn’t see evidence of:
-
whole team having agile ways of working. Whilst the supplier indicated they were following the Service Standard, it wasn’t clear if there was cohesion between them and the Civil Servants who represented team roles.
-
clear prioritisation driven by civil servant product or service ownership leads.
8. Iterate and improve frequently
Decision
The service was rated red for point 8 of the Standard.
During the assessment, we didn’t see evidence of:
- User-led design choices. Instead it felt clear that the delivery was driven by contractual arrangements and end dates.
9. Create a secure service which protects users’ privacy
Decision
The service was rated amber for point 9 of the Standard.
During the assessment, we didn’t see evidence of:
-
the cookie banner should be present when the service is publicly accessible to let users accept or reject any cookies that are not essential to providing the service.
-
where the team had included the use of Google Analytics into their DPIA. The team should include a DPIA to ensure they are securely dealing with this data. See 9. Create a secure service which protects users’ privacy - Service Manual - GOV.UK
-
where the team has obtained SIRO sign off for the use of Google Analytics as described in point 9 of the Service Manual
10. Define what success looks like and publish performance data
Decision
The service was rated green for point 10 of the Standard.
11. Choose the right tools and technology
Decision
The service was rated green for point 11 of the Standard.
12. Make new source code open
Github URL: https://github.com/cabinetoffice/evaluation-registry
Decision
The service was rated amber for point 12 of the Standard.
During the assessment, we didn’t see evidence of:
- The code repository for the service is still private. This needs to be made public for transparency and to provide support to other departments that use the Django framework for their service.
13. Use and contribute to open standards, common components and patterns
Decision
The service was rated green for point 13 of the Standard.
14. Operate a reliable service
Decision
The service was rated green for point 14 of the Standard.
Next Steps
In order for the service to continue to the next phase of development, it must meet the Standard and get CDDO spend approvals. The service must be reassessed against the points of the Standard that are rated red at this assessment.