The Multiplication Tables Check beta assessment
The report from the Beta assessment for the Department of Education's Multiplication Tables Check on the 26th of July 2018.
| From: | Central Digital and Data Office |
| Assessment date: | 26 July 2018 |
| Stage: | Beta |
| Result: | Met |
| Service provider: | DfE - Standards and Testing Agency |
The service met the Standard because:
- they demonstrated a well developed understanding of their different user groups and their needs
- have good ways of working and are empowered to iterate and improve the service based on the needs of users
- have regularly iterated and improved the service during private beta based on feedback from testing with users and the recommendations in the previous Alpha report.
About the service
Description
The Multiplication Tables Check (MTC) is to deliver the first digital assessment for primary aged pupils in year 4 (i.e. aged 8-9 years) in state-funded schools in England.
The MTC check will assess if pupils (year 4) can fluently recall answers to questions from each of the 12 times tables, and will be the first primary KS2 assessment delivered online / onscreen in England. The MTC can only be delivered onscreen due to the fact that the check is timed, with check (as currently configured) comprising of 25 questions, with 6 seconds allowed for each question, and therefore cannot be delivered as other paper based KS2 assessments, no can there be any comparable off-line alternative.
Alongside the actual check for pupils, there is a separate administration area for schools to go to manage pupil registers, user access arrangements for pupils with special educational needs and disabilities, complete a declaration of compliance, see pupil results. STA Service leads will also have access to the administration application to manage the service and extract MI through third party service such as Power BI and Application Insights.
Detail
User needs
In general, the panel was very impressed by the depth and breadth of user research carried out for Beta across the different audiences for the assessment. The team was very clear about their approach, rationale, and use of evidence, as well as their close links to policy and STAs psychometrics team. They showed how they had iterated designs over time, and how their research has challenged their thinking. It was particularly good to see design development being consistently linked back to user needs; and to see the wide geographical spread of schools included. The panel would strongly encourage the team to apply this thinking and way of working as they develop the full end-to-end-service (including set-up, results provision, and user feedback).
The user researcher has conducted a substantial amount of work with schools/pupils who have additional needs, to ensure that the test that can be successfully taken by pupils with a very wide range of needs. The team is to be commended for the effort put into this, and the high level of knowledge that they have now built up in this area.
The panel would encourage the team to share their learning on accessibility with the wider government digital community, through blog posts or presentations. The user researcher has already presented the team’s approach at a recent cross-government Accessibility meetup. The team needs to make sure that this deep knowledge and expertise is not lost during the planned team restructure.
The panel notes that there are some elements where solutions that meet user needs (such as the child-friendly simple URL) are encountering some resistance, and suggests continuing to work on this, gathering evidence to demonstrate the impact on pupils and teachers, and exploring alternatives.
Team
The team went into good detail around their make-up and team process. The team is leading the way for agile delivery within their organisation and are strong advocates of ‘actionable agile’. The team is therefore well placed to influence the working culture and practices within the STA organisation.
Members of the team are based in different locations around the country. They catch-up and do stand-ups daily via online methods and collaboration tools such as Zoom. They also all meet face to face every 2 week sprint.
The panel have some worries that the team will not be best placed to improve the service after the 2019 trial period due to the significant nature of the changes in team make-up due to many of the long term contractors leaving. The team advised they are going to go through a process of upskilling permanent staff so the vast knowledge built up is not lost when some of the Cadence contractors leave the team. It’s very important that STA - retain a strong user centred design core within the team so they can continue to improve the service.
Technology
The Team demonstrated at this Beta Assessment that they had successfully addressed the following technical improvements that they were asked to do to at the outcome of the Alpha Assessment: Firstly they now have a full automated end to end, one click testing and deployment environment for delivering code for this project. Travis, Circle CI, Codacy and BrowserStack are used to support this. The Team have to use JavaScript to support the check’s timed testing functionality but have researched this thoroughly to try to find viable alternatives, though none were forthcoming. They have also stuck to using the mandated long gov.uk URLs as on balance they considered these were better option than alternatives such as BitLy URLs.
The service has been made resilient with a clear action plan should it go offline, through the use of Read Only databases and Geo Replication. Upon going offline, the service can be redeployed to each school within 15 minutes (with associated Census data taking 10 minutes to redeploy).Overnight performance tests are also used to ensure resilience. The Team adhere to open standards through making use of GitHub as an open code repository and their work has already been made use of by another project, namely DfE Sign In. The team have also tested their service works end to end extensively in multiple browsers and platforms. This will prepare them well for deploying the Multiplication Tables Check to be a web app, accessed on mobile platforms were this to become a future requirement.
The team ran three sets of trials on the Multiplication Checker, those in Feb and March used anonymised data, those in June using real pupil data. SQL Data Masking was used to enable the development team to work with the data without pupils’ personal details needing to be made visible to developers. The Team has also carried out due diligence to ensure their use of data is GDPR compliant. The service’s threat model has been designed to follow NCSC security guidelines , which has involved using the NCSC Web check service as well as the DSAM service penetration testing, Content Security Policy, internal overnight security and uses TLS 2 protocol as its baseline standard.
Design
The team have done some excellent work iterating and improving the service since their last assessment. They presented these changes in a very clear manner and explained the user needs behind the changes. The service has three main user groups but the main focus for the team has been the teacher admin tool and the student test itself. The team went into good detail about the extensive research they have done with both groups.
The team explained how most users get through the various service journeys first time but are well aware of the areas that need attention. Regarding the tables check this is notably around URL input for students (with the main cause of drop-outs being the long gov.uk mandated URLs not being entered correctly by 8 or 9 year olds). They have been engaged in conversation with gov.uk for some time around the prospect of a short URL so students can access the tables check more easily. The team were also asked whether they should use ‘bit.ly’ links if they can’t get sign off for short gov.uk URLs. The teams reasons for not using ‘bit.ly’ links was down to a perceived lack of trust in them. The panel feels this should be looked into further to see if there is actually a lack of trust in them.
Once users are on the check home page, however, success rates of authentication have averaged out at over 91% over the three 2018 trials. This is a substantial increase from the June 2017 private beta trial which achieved a 74% success rate with pupils who were two years older. Since the policy change from year 6 to year 4 pupils in September 2017, the service team have undertaken a significant amount of user research with this new age group to ensure the school password format is recognisable and can be easily input by this year group.
The team were questioned about the 6 second question interval. They are confident this is an appropriate amount of time due to the nature of the check being around fluent recall; specifically the March 2018 trial ran the check at three different timings with this age group. Data gathered by the pupil check has ensured that the test validity can stand up to scrutiny from regulators. When questioned whether or not time should be extended for specific Special Educational Needs pupils the team said this was something they were continually looking into. However, there are already a wide variety of modifications being prototyped and tested with users in this age group to make the check as accessible as possible without the need to alter the time which would have a negative impact on test validity.
The lack of content design expertise in the team has lead to some questionable areas of content design. Although this is undergoing improvement the team need to be aware and engage with content design more often, rather than their current process of content reviews every few months.
Analytics
The team had an initial conversation with the Government Digital Service about the Performance Platform, but they need to make it a priority work with the Performance Platform team to build a dashboard and ensure they have processes in place to keep it up to date. We understand that due to the nature of the service data will only be provided to the platform annually covering a 3 week period and that this can only take place after the trial period (circa summer 2019). The team understand the importance of uploading their data to the Performance Platform and have begun preparations for their dashboard, agreeing internally which metrics they will be submitting and where the data for these will be sourced from.
The team discussed potentially moving away from Google Analytics as they can get the data they need from other sources - this includes rich insight into device usage which helps the team understand how the service is being accessed. However, while Google Analytics remains on the site the team need to urgently ensure they are anonymising the user IP address, and that they have opted out of data sharing with 3rd parties - there is advice in the service manual under information security and privacy which should help.
The team are making good use of multiple data sources and have made provisions to collect additional feedback from users through smart survey and are investigating other options too. The team also have access to communications resource who will monitor the response to the service on social media during the voluntary period in 2019.
Application Insights is being used to actively monitor service performance on the Azure Cloud hosting platform - with service manager designated having direct access to dashboards as schools trailed the service.
Additionally the service team are using Power BI to extract data from the Azure SQL database to obtain the management information service managers need to run the service within STA. The Power BI tool rolled was rolled out in a way that provided internal stakeholders with quick access to daily data throughout trial periods, and further dashboards will be set up for service managers when the check is rolled out nationally. The team have a plan in place to run internal training on the tool to give more users access to the data.
The team offered examples of how they use these two reporting tools to make changes to the service, for example identifying issues around page load and subsequently adding this to their backlog. When the team do make changes to the service they actively measure these to confirm their assumptions, they demonstrated this to the panel by explaining improvements to the percentage of successful user logins.
Recommendations
To pass the next assessment, the service team must:
- engage fully with the Performance Platform team by confirming the dashboard structure and agreeing when it will be updated (must be done prior to the trial period)
- build the ‘familiarisation’ area for pupils (must be done prior to the trial period)
- have a detailed accessibility audit and have acted on it’s recommendations (must be done prior to the trial period)
- keep a strong user centred design unit within the team, so they can continue to improve the service after the trial period (i.e. User Researcher, Designer and Content Designer)
- ensure proper knowledge sharing has taken place when contractors leave the team and that they are sufficiently replaced by new team members
- continue to develop and test modifications to the pupil check or amended for different Special Education Needs pupils to ensure all reasonable adjustments are being made to meet the requirements of the Equalities Act; while ensuring these checks to not compromise the validity of the assessment
- have resolved how they are going to handle the issuing of check results in a way that meets user needs.
The service team should also:
- research with users whether bit.ly links actually effect the trust in the service for users
- examine different ways of designing the teacher admin dashboard to continue to ensure it’s easy for teachers to administer the checks to pupils
- have more regular input from a Content Designer.
Next Steps
You should follow the recommendations made in this report before arranging your next assessment.
This service now has permission to launch on a GOV.UK service domain. These instructions explain how to set up your *.service.gov.uk domain.
Submit feedback
Submit feedback about your assessment.
Get advice and guidance
The team can get advice and guidance on the next stage of development by:
- searching the Government Service Design Manual
- asking the cross-government slack community
- contacting the Service Assessment team