Take an online test when I apply for a job in the Civil Service beta reassessment report
The report from Cabinet Office's Take an online test when I apply for a job in the Civil Service beta reassessment on 27/01/21
Digital Service Standard assessment report
Take An Online Test When I Apply For A Job in the Civil Service
| From: | Central Digital and Data Office |
| Assessment date: | 27/01/21 |
| Stage: | Beta Reassessment |
| Result: | Met |
| Service provider: | Cabinet Office |
Previous assessment reports
Alpha assessment report: Met (13 August 2018)
Beta assessment report: Not Met (25 March 2020)
Service description
The Online Take A Test When I Apply For A Job in the Civil Service also known as Online Tests and Assessments (OTAP) service is part of Government Recruitment Service (GRS), and provides a SaaS platform to develop psychometric tests for recruitment. The tests are developed in-house by the Tests and Assessments (OTAP) team, which includes GRS Occupational Psychologists and a multidisciplinary digital team. Tests are made available by integration with CSJobs, a wider SaaS platform where Recruiters set up vacancies and manage candidate applications, and where job seekers can find and apply for jobs in the Civil Service.
When they advertise a vacancy, Recruiters and Vacancy Holders have the opportunity to add our Online Tests as part of their selection process, alongside other methods like CV and personal statement. For applicants, this means that if a job is set up to require an Online Test, they will need to undertake this as part of their application. The service allows Applicants to complete tests which accurately reflect their ability in a fair and inclusive way, contributing to a more efficient and transparent recruitment journey.
Service users
● applicants - users who are applying for a job in government. They want the role and have often already invested time and effort into researching and applying
● vacancy holders - users who have a vacancy in their team / area that needs to be filled with a good candidate
● recruiters - users who are specialists in Government Recruitment Service (GRS) or department recruitment that advise and support vacancy holders through the recruitment process
● test creators - specialists in GRS or department Human Resources that develop tests and test content (items), typically Occupational Psychologists and Psychometricians
12. Make sure users succeed first time
Decision
The service met point 12 of the Standard.
What the team has done well
The panel was impressed that:
- the service team responded to the assessment feedback and focused on the recommendations and actioned items highlighted in the accessibility report
- the service team had focused their performance data to understand user pains through extensive performance data
- tested the journey and considered accessibility needs to make sure the test could be performed
- hired an interaction designer, content designer and had made changes to improve usability of the service. Pleased the team took this action away and fixed experience issues highlighted in the previous assessment
- completion rates were used to evidence ‘success first time’ from a quantitative perspective. This was complimented with thematic analysis of those who both completed and failed to complete. This data was used to identify areas for improvement, which the team showed 2 iterations. The first iteration showed clear improvement and the second iteration was successfully released but needs more time to confirm improvement significance
What the team needs to explore
Before their next assessment, the team needs to:
- as an advisory: the journey involves a 3rd party journey so using funnel visualisation and splitting the journey into segments, therefore using multiple completion rates may help to visualise the risk points in the journey. There was evidence of these types of metrics but without the funnel visual, this message can often get lost when looking at larger dashboards
- send over examples of alerts shown to users when there is a problem to help users feel at ease. This wasn’t demonstrated during the assessment but we were told we can see an example
- service team focused on applicants completing the test. Would have liked to have seen some examples to show efforts to demonstrate how the service team have carried out work to demonstrate the vacancy holder understands which test to include. Guidance material was requested
- when a user is stuck or requires support….how support can be requested during any time of the test. (Example to follow post assessment)
14. Encourage everyone to use the digital service
Decision
The service met point 14 of the Standard.
What the team has done well
The panel was impressed that:
- the team had taken on board the various concerns and critique surrounding accessibility issues and taken major steps to understand and address the needs
What the team needs to explore
- purely as a note, the latest accessibility audit demonstrated that the service had a small number of issues still to address which the team had already triaged as minor and that could be addressed (mostly) internally with a single issue requiring supplier intervention which had been arranged
15. Collect performance data
Decision
The service met point 15 of the Standard.
What the team has done well
The panel was impressed that:
- the team has a large number of data sources feeding the dashboards, which are predominantly housed in data studio, powerBI and excel. The reporting tends to stay within the development team so access and interpretation is not an issue, however mention was made about wider stakeholders potentially receiving slimmed down reports/boards
- the key KPI’s were developed using a mix of GA and direct from the supplier data dumps. The team therefore has the ability to interrogate and test GA data for accuracy, this is a vital part of data accuracy testing considering the google analytics nuances and the cookie policy impact
- measures include journey KPI’s as well as statistical dissection of the performance and success of the tests which clearly showed that the team are in a position to not only monitor and identify areas of interest in journey flow and user behaviour online, but also where questions have a potential bias or weakness especially when cohorting by user characteristics. This was a very welcome addition to the data already shown
What the team needs to explore
Before their next assessment, the team needs to:
- the team made mention of tracking returning users and multiple attempts as a measure of success. This is a challenge as the tracking requires the 3rd party provider to use GA or similar analytics strategies for which they are limited due to outside influence. This would be a very useful measure as it would compliment and provide more detail behind completion rate trends.
16. Identify performance indicators
Decision
The service met point 16 of the Standard.
What the team has done well
The panel was impressed that:
- the team has a comprehensive performance framework which has been mapped against the user and business needs which has meant the team has focused dashboards/reporting and conversations around iterating/performance and evolving the service to better meet user needs
- the team has developed baselines, targets and thresholds for its main KPI’s. When thresholds are breached, the team illustrated the process to investigate and develop solutions quickly. The example given was server downtime and how quick the team identified, reacted to and resolved the issue to return to BAU
- the team mentioned weekly meetings used to discuss areas of interest and monthly boards also produced to update stakeholders. A clear process was shown how the KPI’s play an important role in understanding service performance trends and areas of the service that need further investigation
What the team needs to explore
Before their next assessment, the team needs to:
- the boards shown are comprehensive. The team may want to look at creating a set of themed boards and reducing the board elements as they engage with the wider stakeholder groups to ensure interpretation of performance is clear, while also ensuring that if a measure were to see a drastic change, that it is not lost in a large number of visuals. However this is very much down to viewers ability to digest large numbers of different data visualisations so this is an advisory rather than a must do