Research and analysis

National Reference Test 2021: Performance Across Content Areas

Published 20 December 2022

Applies to England

The NRT is uniquely placed, as a Key Stage 4 assessment which directly measures performance over time, to contribute to the evidence on the impact disruption due to the Covid-19 pandemic. The 2020 NRT administration was largely unaffected by the pandemic, as the testing window ended before schools were closed for the first national lockdown. We can assume, therefore, that the outcomes up to, and including, 2020 for the NRT provide a relatively precise and stable measure of performance of the population over time. Accordingly, the NRT outcomes in 2021 have the potential to provide useful insights into the way in which student performance may have been impacted by the pandemic.

The main NRT outcomes, reported in the 2021 Results Digest, show overall performance on the NRT in English and mathematics over time indicating how the subjects may have been impacted as a whole. This study provides additional analysis of the NRT 2020 and 2021 data to explore whether different areas of the curriculum have been differentially impacted by the disruption to teaching as a result of the Covid-19 pandemic. This may highlight areas that were more difficult to learn at a distance, were given a lower priority than others or required more explicit teaching of concepts and/or skills.

The 2021 NRT results indicated that there was no significant change in the overall performance in English for the 2021 cohort of students compared with the 2020 cohort. The absence of a global difference could mask relative changes for different parts of the curriculum (for example, it is possible that we might have seen improved performance in writing with decreased performance in reading). The analyses in this study compared relative performance between reading and writing and their composite assessment objectives. No evidence of any clear differences across the curriculum areas was found, suggesting that performance on the NRT in 2021 was comparable to that of 2020 across all areas of the English curriculum assessed in the NRT.

The 2021 NRT results indicated that there was a significant drop in performance in mathematics for the 2021 cohort of students compared with the 2020 cohort. This study investigates further whether this drop in performance is a result of larger decreases in performance in specific areas of the mathematics curriculum, or whether it reflects smaller decreases in performance across all curriculum areas. The evidence from these analyses indicates that performance fell by a similar amount for all areas of the curriculum in mathematics between 2020 and 2021 and that the disruption caused by the pandemic therefore did not have a differential impact across the mathematics curriculum.

As the extent of Covid-19 pandemic-related disruption was variable at the school, class and even student level, then the absence of a clear pattern in changes in performance at the curriculum level for English and mathematics is perhaps unsurprising.

1. Background

As an independent measure of performance over time of a nationally representative sample at GCSE level, the NRT outcomes have the potential to provide some useful insights on the impact of disruption to education due to the Covid-19 pandemic.

The NRT was designed to routinely report outcomes each year in terms of the proportions of students achieving at or above three key GCSE grades, defined as being equivalent to the ability required to achieve those grades in 2017 (the baseline year). The 2020 NRT administration was largely unaffected by Covid-19, as the testing window ended before schools were closed for the first national lockdown^{[footnote 1]}. We can assume, therefore, that the outcomes up to, and including, 2020 for the NRT provide a relatively precise and stable measure of performance of the population over time. This should allow us to identify significant changes in performance in 2021 which are likely to be attributable to the impact of the Covid-19 pandemic^{[footnote 2]}. The eight test booklets that make up the NRT had the same content in 2020 and 2021, which means that direct comparisons between the performances of the two representative samples are possible.

We might have expected that the performance of the 2021 sample would be lower than for the previous years given the possible loss of learning due to the Covid-19 pandemic. While the term ‘learning loss’ has attracted some criticism for its negative focus, it has been used to reflect the impact of the pandemic on students’ learning and there is a commonly shared understanding of what it means. Here, it is used to refer specifically to a reduction in the level of attainment in different areas of the English and mathematics curriculum that are attributable to both direct and indirect impacts of the pandemic. This definition is consistent with that used by Ofqual in its review of literature on learning during the pandemic, published last year.

As reported in the National Reference Test 2021 Results Digest, there were, as expected, significant drops in the estimated proportions at or above the ability levels equivalent to grade 4, grade 5 and grade 7 for mathematics. For English, the outcomes were perhaps more surprising: performance was very similar, on average, in 2021 and 2020, with no significant differences at any of the three grade boundaries. Though not statistically significant, there was a slightly higher proportion of students at grade 7 and above in 2021 than in 2020 and a slightly lower proportion achieving grade 4 and above, indicating a slight widening of the ability distribution relative to previous years.

1.1 Overview of NRT

The NRT is a short test which reflects the sorts of questions that are in the GCSE examinations for English language and mathematics, and the questions largely remain the same each year. It is taken by a representative sample of students in year 11 who are taking their GCSEs in the same academic year. The NRT is designed to be accessible to all eligible students. The questions in the English and mathematics tests are aimed mostly at grades 3 to 7 but also include some questions aimed at grades 1 and 2 and grades 8–9 to help make them accessible and / or challenging to all students.

For each subject, the questions are arranged into eight test booklets of 50 marks. Test questions are grouped into blocks and each test booklet is made up of two blocks of questions. The blocks are organised in an overlapping design such that each block, and therefore each question, appears in two test booklets. This design provides a broad coverage of the curriculum and good measurement of the national population, while minimising the time each student is tested. Each student has one hour to complete either one English or one mathematics booklet.

The English NRT comprises a reading and a writing component. Each component is marked out of 25 marks and students are advised to spend broadly equal time on each. The majority of the marks for reading and all of the marks for writing are for extended response questions. The remaining marks on each reading test are for short-answer responses. The English NRT covers all of the GCSE assessment objectives for reading and writing (excluding those for spoken language).

The mathematics NRT covers the full range of subject content for the GCSE specification (number, algebra, ratio and proportion, geometry and measures, and statistics and probability) and the assessment objectives (using and applying standard techniques; reason, interpret and communicate mathematically; solve problems within mathematics and in other contexts). It is not a tiered assessment. Most of the questions are based on the GCSE content that all GCSE mathematics students have to study and includes some more straightforward questions at the start to help encourage students. A few questions drawing on higher tier content are at the end of each test booklet.

More information on the NRT can be found in the General Report.

The NRT was constructed as a test to report on the performance of a student cohort, in English language and mathematics, between years. During test construction, the curriculum characteristics for every question in the NRT were recorded to ensure that the curriculum balance of the NRT matched that of the GCSEs to allow for a valid comparison to GCSE performance. For the purposes of this study, we can use these designations to explore changes in performance across the different areas of English and mathematics between 2020 and 2021. Specifically, these are the different assessment objectives relating to reading and writing for the English curriculum and the subject content areas and assessment objectives for the mathematics curriculum.

It should be recognised that for the NRT 2021 cohort, both years of Key Stage 4 have been defined by high levels of disruption, including long periods of school closures during the two national lockdowns as well as periods where Covid-19 cases in bubbles or schools resulted in some students spending additional periods of time away from the classroom. However, it is also important to recognise that since the start of the Covid-19 pandemic this particular cohort, compared with other year groups, have had extended periods of time in school participating in face-to-face lessons and potentially catching up on learning disrupted earlier in the pandemic. This complex picture highlights the fact that individual students, cohorts and schools may have experienced different levels of disruption caused by the pandemic.

As an ‘exam’ year, the NRT 2021 cohort were one of the priority year groups to return to school at the end of the first national lockdown in June 2020. Although this meant that these students may have had an additional six weeks of face-to-face teaching compared with other year groups, provision of lessons and student attendance may not have been consistent across the cohort. The 2020/21 academic year began relatively normally with all students returning to school at the start of the autumn 2020 term. This normality was short-lived in some regions however, as many students experienced further disruption during this period with bubble and year group closures, as well as time away from schools for self-isolation due to high Covid-19 rates. The second lockdown, which began in January 2021, saw most students experiencing another long period of remote learning and, for the NRT 2021 cohort, the announcement that formal GCSE examinations would not take place that summer. The vast majority of students returned to school in March 2021 and therefore the NRT students spent four weeks participating in face-to-face lessons before the NRT 2021 was administered. However, the cancellation of formal GCSE exams may have meant that these students were working in a different way or completing different sorts of assignments (for example internal assessments) in the weeks preceding the NRT administration. Therefore, in some respects, these students may have been more ‘test ready’ compared with the students who participated in the NRT in 2020. However, they may also have been less motivated as they were not preparing for formal examinations, as would normally be the case. This, as well as the two-month delay to the administration of the NRT 2021, may have impacted student performance.

It is not possible to say definitively how students’ learning experiences during this period may help to explain the findings from this additional analysis. However, Ofqual have undertaken further analysis of the contextual information collected from schools and students during the administration of the 2021 NRT. The findings from Ofqual’s analysis may provide some useful context in the interpretation of the findings from this study.

1.2 Interpreting the data

The following points should be borne in mind when comparing student performance on the NRT in 2020 and 2021 and interpreting how these differences reflect Covid-19 disruption:

The administration of the NRT in 2021 took place approximately two months later than would usually be the case, due to the closure of schools to most students in the spring term of 2021.

The administration period was relatively soon after the return to school for most students, so some students may still have been adjusting to being back in school.

GCSE examinations had been cancelled prior to the administration of the NRT in 2021; this was not the case for the 2020 administration, even though they were later cancelled.

The administration took place at a time when teachers were making judgements for their teacher-assessed grades for GCSEs, meaning some students may have been more ‘test ready’ than they would ordinarily be.

2. Are different areas of the curriculum impacted differentially?

The aim of this research study was to see whether there is evidence from the NRT that different areas of the curriculum have been impacted by the pandemic.

The main NRT outcomes show the overall performance for the NRT over time in English and mathematics and gives some indication as to how performance has changed on each subject as a whole. The additional analysis in this report aims to show if there are areas of the curriculum which have been impacted more by the pandemic than others. This may highlight areas that were more difficult to learn at a distance, were given a lower priority than others or required more explicit teaching of concepts and/or skills.

The NRT has been designed to provide a global quantitative measure of performance in both English and mathematics. In order to provide a valid comparison with the GCSEs, the NRT has been constructed to provide a similar distribution of assessment objectives and subject content. Therefore, within the NRT we can analyse the performance of students in the different specified areas of the curriculum included in the NRT and compare these to the performance in the 2020 NRT administration. Given that the NRT in 2021 had the same questions as the NRT in 2020, it provides a valid measure of differential performance between cohorts for the different curriculum areas.

The English NRT is divided into the two main subject areas of reading and writing. Each of these subject areas is worth 50 per cent of the marks of the English NRT. An initial comparison can be made between the performance of the 2020 cohort and of the 2021 cohort in these two subject areas. This comparison will test the hypothesis that the students’ performance in reading and writing has been impacted by the pandemic.

Within these subject areas, more granular detail of potential differences can be derived from looking at the different assessment objectives. The English curriculum is divided into six assessment objectives (AOs). Four of these (AO1–4) refer to reading skills and two to writing skills (AO5–6). In both cases there is a progression of skills (see Table 2.1) through these AOs which means that analysis of them may identify relative changes in the performance of students related to the level of demand of the items.

Each AO for English is represented by at least 20 marks in the NRT so there are sufficient marks for differences in performance of all AOs to be identified. An analysis comparing the change in performance of the English AOs in the 2021 cohort will test the hypothesis that some of the skills may have been differentially impacted by the pandemic. For example, it is plausible that the available time had been focused on identifying and explaining language in texts and proportionally less time to the higher order skills of comparing and evaluating.

2.1 Table 2.1: Assessment objective coverage for English

Assessment objective	Description	Number of marks in the NRT
AO1	Identify and interpret explicit and implicit information and ideas. Select and synthesise evidence from different texts.	20
AO2	Explain, comment on and analyse how writers use language and structure to achieve effects and influence readers, using relevant subject terminology to support their views.	40
AO3	Compare writers’ ideas and perspectives, as well as how these are conveyed, across two or more texts.	20
AO4	Evaluate texts critically and support this with appropriate textual references.	20
AO5	Communicate clearly, effectively and imaginatively, selecting and adapting tone, style and register for different forms, purposes and audiences. Organise information and ideas, using structural and grammatical features to support coherence and cohesion of texts.	60
AO6	Candidates must use a range of vocabulary and sentence structures for clarity, purpose and effect, with accurate spelling and punctuation.	40

The mathematics curriculum, and the structure of the NRT, covers five main subject areas as shown in Table 2.2. It is possible that, as a result of the disruption to teaching caused by the pandemic, there may have been a change in emphasis given to the teaching of different subject areas compared with previous years. For example, there may have been an emphasis on the core subject area of number at the expense of a more involved subject area such as algebra. The NRT data allows us to investigate the hypothesis that disruption has differentially impacted these subject areas in mathematics.

2.2 Table 2.2: Subject content area coverage for mathematics

Table notes:

Each item is written with a specific subject intent, for example, to cover algebra. This distribution is reported as the number of marks associated with items for each subject area. However, each mark that makes up an item will also have a specific subject statement allocated for the specific requirement of that step in the question and this may differ from the subject area for the item. This is reported as the number of individual marks in the NRT. The latter provides a more accurate picture of the balance of subject content across the NRT as a whole.

Probability and statistics are separate areas in the curriculum but are considered together for the purposes of the subject content coverage. In some of the analyses in this study the performance of probability and statistics are considered separately.

Mathematics subject content area	Number of marks associated with items for each subject content area in the NRT	Number of individual marks in the NRT
Number	39	46
Algebra	53	49
Ratio, proportion and rates of change	34	38
Geometry and measures	46	39
Probability and statistics	28	28

As with English, the mathematics skills are also assessed using an AO for both GCSE and NRT. The distribution of marks awarded for each AO in the NRT is presented in Table 2.3.

2.3 Table 2.3 Assessment objective coverage for mathematics

Assessment objective	Description	Number of marks in the NRT
AO1	Use and apply standard techniques	85
AO2	Reason, interpret and communicate mathematically	59
AO3	Solve problems within mathematics and in other contexts	56

3. Sample

The achieved sample sizes for the NRT for the years in this study are shown in Table 3.1. Fewer schools took part in the NRT in 2021 than in other years following the postponement of the administration period to the summer term.

3.1 Table 3.1: School and student achieved samples NRT 2020 and 2021

Achieved sample	2020	2021
English achieved school sample	332	214
English achieved student sample	6,639	4,030
Mathematics achieved school sample	333	216
Mathematics achieved student sample	6,756	4,143

A bias analysis was carried out to check that the reduced sample was still representative in terms of the stratifier used to select the NRT sample: GCSE performance at school level in 2019. A number of other school factors which could have affected performance due to the impact of the pandemic were also investigated. Therefore, the sample has been broken down further by the proportion of students eligible for free school meals (FSM), geographical region and absence rates. While the data shows that the NRT samples do not perfectly represent schools in terms of the proportion of students with FSM and geographical region, they do indicate that there has not been a major change compared with 2020. Analysis of the absence rates during the autumn term 2020 found that there were no significant differences in the absence rates during this term between those schools that participated in NRT 2021 and those schools which were invited but were unable to participate due to the pandemic. Therefore, the evidence suggests that despite the smaller sample size, comparing performance in 2020 and 2021 can still be considered valid. However, when interpreting the findings of this study it should be borne in mind that there is no direct measure available of how individual schools coped with the disruption of the pandemic, and therefore we cannot rule out the possibility that the schools that struggled more with the impact may have been more likely to withdraw from the NRT.

The number of students completing each of the NRT booklets for English and mathematics is presented in the tables 3.2 and 3.3. The fewer completed booklets in 2021 reflects the fact that fewer schools participated in the NRT in 2021. However, the design of the assessments means that the number of students completing each booklet is similar within each year.

3.2 Table 3.2: Number of completed test booklets in English

Booklet	Number of students completed tests NRT 2020	Number of students completed tests NRT 2021
Booklet 1	834	492
Booklet 2	838	499
Booklet 3	827	495
Booklet 4	835	507
Booklet 5	840	505
Booklet 6	832	507
Booklet 7	819	518
Booklet 8	814	507
Total	6639	4030

3.3 Table 3.3: Number of completed test booklets in mathematics

Booklet	Number of students completed tests NRT 2020	Number of students completed tests NRT 2021
Booklet 1	844	513
Booklet 2	832	518
Booklet 3	841	513
Booklet 4	866	520
Booklet 5	841	526
Booklet 6	839	514
Booklet 7	857	513
Booklet 8	836	526
Total	6756	4143

4. Methodology

All of the data presented in this analysis are derived from Classical Test Theory (CTT) analysis which is based on the raw scores achieved by students from items for the different curriculum areas. For the main analysis of the NRT, Item Response Theory (IRT) is the preferred methodology for comparing the performance of different cohorts of students between years. Given the connected design of the NRT booklets, IRT allows the performance of all students to be compared in a meaningful way, irrespective of the specific items that they sat. For the performance of items themselves, CTT provides a more direct comparison and is the method used to compare item performance between years. As the items in the booklets were the same in 2020 and 2021 then they can be considered as parallel tests and as such, CTT analysis provides a valid comparison between years. Given the validity of this approach and the more straightforward interpretation of CTT statistics for item performance, this approach has been taken in this study. Further information on the technical terms included in the methodology can be found in the Glossary of this report.

A summary of the analyses presented in this study is provided below.

1. Profile plots. Profile plots (Bechger and Partchev, 2021) are very useful for investigating whether two groups of students obtain the same test score in the same way. The profile plot maps the most likely combination of scores on one curriculum area compared with the rest of the test for each overall total score on the booklet. The plots are presented at the booklet level so that data for each curriculum area based on the same set of students (that is, the ones that completed that booklet) are not complicated by student specific factors. The profile plots are presented together for both the 2020 and 2021 cohorts. Any large differences in performance in a curriculum area would be noted as a deviation of the plots for the two years. A consistent deviation would be noted as occurring for the same curriculum area in the different booklets.
2. Booklet scores. These graphs present the proportion of the total booklet scores allocated to each curriculum area. The curriculum areas are presented in different colours, and graphs are presented as cumulative bar graphs. As with the profile plots, the graphs are presented at the booklet level so that student-specific factors do not form part of the analysis. Note also that the graphs are presented proportionally so that differences in the total scores per booklet between years are not presented. This format allows for a comparison of the relative proportion of scores allocated to each curriculum area between years.
3. Facilities at the item level. The facilities show the proportion of students who answered a particular item correctly. These graphs present the facility for each curriculum area in each booklet between 2020 and 2021. This provides a more obvious way of comparing the direction of change in facilities for curriculum areas than the booklet score graphs.
4. Item facility plots for English. These graphs present the facilities for items in the analysis in 2020 compared with 2021. The items are separated with each curriculum area plotted on a separate graph. This analysis presents global information on the changes in facilities between these years rather than just at the booklet level. The graphs include a diagonal line indicating equivalence of scores between the years. Items where the facility increased from 2020 to 2021 are plotted above this line and items where the facility decreased are plotted below the line. The data from these plots are provided in Appendix B1 Table B1 and were used for statistical t-testing for significant differences in assessment objective performance between 2020 and 2021 (Table 5.1).
5. Sub-item facilities for mathematics. For mathematics, the items are marked based on achievement of students on each individual mark (of the 200-mark total). As each mark has a designated subject content area and assessment objective, then analysis at the sub-item level allows a more accurate analysis of the different curriculum areas than at the item level. This is because most items in mathematics include a mix of different subject areas and assessment objectives, which does not occur for English. In order to analyse performance across the different curriculum areas between the years, the change in facility between 2020 and 2021 is averaged for all subject areas. To account for changes in the relative difficulty of items in different curriculum areas, proportional changes to facilities (change in proportion correct divided by 2020 proportion correct) are also compared. The data used for these analyses are provided in Appendix B2 Table B4 and the results of the analysis are presented in Table 5.2. These data were also used for statistical t-testing for significant differences in assessment objective performance between 2020 and 2021 (Table 5.3).
6. Sub-item facility rankings for mathematics. The sub-items for 2020 and 2021 were ranked in order from the lowest facility (most difficult item) to the highest facility (easiest item). This provides 200 data points for each year. The change in ranking for each sub-item was calculated and these were averaged across the different areas of the curriculum. This allows the relative change in performance for each area of the curriculum to be measured against changes in the other areas. A negative number indicates that the curriculum area has performed comparatively less well than the other curriculum areas. The analysis of facility rankings is provided in Table 5.2.

5. Outcomes

5.1 English

Reading versus writing: profile plots

Initial analysis for English was to look for any evidence of specific changes in relative performance for items targeting reading skills compared with items that target writing skills. This represents the main division in English booklets and is a valid comparison to make as each component makes up half of the marks for each booklet.

The profile plots map the most likely scores for the reading component of each booklet (horizontal axis) against the most likely scores for the writing component (vertical axis) for students with the same total score (the diagonal lines). The reading vs writing profile plot for booklet 1 is shown in Figure 1. The 2020 cohort is presented by the red line and the 2021 cohort by the blue line. We can see that, for example, for a total score of ten (the diagonal line ending with a small number 10), the 2020 sample were most likely to score four marks on reading and six marks on writing; whereas for students in the 2021 sample also scoring ten in total, the most likely combination of scores was six on reading and four on writing. The equivalent profile plots for all booklets are presented in Appendix A1 Figure A1. The plots for some of the booklets show a similar pattern of performance as that seen in the booklet 1 plot, that is for the lower half of the score range the 2020 population seems to favour writing and 2021 seems to favour reading. However, these differences are small and they are not consistent across the booklets. Therefore, there is no evidence of a consistent difference in performance between reading and writing in the NRT between 2020 and 2021.+++

Figure 1: Profile plot of scores for reading versus writing scores for booklet 1

Assessment objectives: profile plots

The performance in reading and writing for English can be further broken down by AO with AO 1–4 denoting reading skills and AO 5–6 denoting writing skills (Table 2.1). Profile plot analysis was also performed on the different AOs with the score for each AO plotted against the score for the remainder of the booklet.

The profile plots for the reading AOs for booklet 1 are shown in Figure 2. The profile plots for the writing AOs for booklet 1 are shown in Figure 3. The equivalent profile plots for all booklets are presented in Appendix A1 Figures A2 to A7

Although there are minor differences in the profile plots across different AOs, these differences are small and not at all consistent across booklets, suggesting that there are no clear patterns for differences in the way that students obtained these scores between 2020 and 2021. This observation applies to all the English AOs. As such there is no evidence of a clear change in performance of the English AOs from 2020 to 2021.

Figure 2: Profile plot for reading assessment objectives for booklet 1

Figure notes: All booklets contain marks for AO1 and AO2. Booklets 1,6,7 and 8 include marks that cover AO3 skills. Booklets 2,3,4 and 5 include marks that cover AO4 skills.

Figure 3: Profile plot for writing assessment objectives for booklet 1

Assessment objectives: proportions, facilities and mean scores

As the booklets for 2020 and 2021 contained the same items, we can compare the scores within those booklets directly. Figure 4 shows small differences for some proportions of certain AOs between 2020 and 2021. However, these differences are not substantial and never consistently favour a certain year.

Figure 4: Proportions of booklet scores by assessment objectives for 2020 and 2021

This same observation is true for the mean facilities of the different AOs in 2020 and 2021. The full overview of the graphs is in Appendix B1 Figure B1, but we can give an example in Figure 5. For instance, the data for booklet 1 show a slight increase for AO2, but for booklet 3 there was a decrease for AO2.

Figure 5: Mean facilities for assessment objectives for booklets 1 and 3

In addition, Table B1 in Appendix B1, with mean scores and standard deviations for the different AOs, does not show substantial differences.

Item facilities comparison plots

Another way to compare the relative performance of AOs between years is to look at the changes in facility for each item covering that AO between 2020 and 2021. This provides a more global analysis than the profile plots as it includes all of the items from the different booklets together. This data is presented as scatter plots in Figure 6.

Figure 6: Item facilities by assessment objective for English comparing 2020 to 2021

This analysis is consistent with the overall outcome of the English NRT in that there is no significant change between the two years. However, there is a suggestion of a small drop in performance for items covering AO1 and AO2, as the majority of these items are placed below the diagonal line. In contrast, all of the items for AO3 to AO6 lie on, or very close to, the diagonal line indicating that they performed similarly between the two years.

A t-test was used to compare the item facilities for AO1 and AO2 between 2020 and 2021, as shown in Table 5.1. In both cases the average difference in facilities was small and not statistically significant (p=0.79 for AO1 and p=0.65 for AO2). This supports the evidence from the profile plots that there were no significant differences in performance for any of the English NRT AOs between 2020 and 2021.

Table 5.1: Comparison of mean facility for AO1 and AO2 items

AO1 items	Mean facility	Standard deviation of facility
2020	0.54	1.45
2021	0.52	1.44

AO2 items	Mean facility	Standard deviation of facility
2020	0.32	0.82
2021	0.31	1.44

Gender

This analysis also explored whether the impact of the disruption of school closures due to the pandemic was the same for the boys and the girls in the NRT sample in terms of their relative performance across the curriculum areas. Whilst there are substantial differences between the performance of boys and girls in the NRT English assessment – girls tend to perform better than boys – there appear to be no notable changes in the performance of boys and girls across the different assessment objectives (AOs) between 2020 and 2021. This is perhaps unsurprising given that across the whole sample very few changes in performance on the different AOs were seen. The figures for this analysis are presented in Appendix C Figure C1.

Conclusion for English

The NRT results from 2021 indicate that there was no significant change in performance in English compared with 2020. This study compared the relative performance by assessment objective in English to see whether any relative changes in performance in particular areas of the curriculum could be discerned that may identify specific English skills that have proved to be difficult to teach or learn, or given different levels of priority as a result of the disruption to the 2021 cohort caused by the Covid-19 pandemic. The analyses of profile plots, booklet scoring and gender factors provide no evidence of clear differences in the performance at the booklet level that might indicate any consistent change in performance across the curriculum. The facility plots across all booklets also show that there were no significant differences in the scores between different AOs.

5.2 Mathematics

Subject content areas: profile plots

The NRT for mathematics is divided into five subject content areas and three assessment objectives as set out in Tables 2.2 and 2.3. Figure 7 shows the representative profile plots for all of the subject content areas for mathematics booklet 1. The equivalent information for all of the booklets is presented in Appendix A2 Figures A8 to A13.

Because of the way that the mathematics curriculum is divided there are many more profile plot comparisons to make. There are few differences between the profile plots for mathematics and, as with the English, there are no consistent differences visible on the profile plots. As such, there is no evidence that performance in the mathematics subject areas differed between 2020 and 2021.

Figure 7: Profile plot of scores for the mathematics subject content areas for booklet 1

Figure notes: Subject content areas on Figure 5.7: A=algebra; G=geometry and measures; N= number; R=ratio and proportion; P=probability; S=statistics. In the NRT subject content distributions probability and statistics are counted together.

Subject content areas: proportions, facilities and mean scores

As the booklets for 2020 and 2021 for mathematics contained the same items, we can compare the scores within those booklets directly. Figure 8 shows only small differences in the proportions of certain subject content areas in mathematics between 2020 and 2021. However, these differences are not substantial and never consistently favour a specific year.

Figure 8: Proportions of mathematics booklet scores of subject content areas for 2020 and 2021

If we take a look at the mean facilities (proportion correct) of the different subject content areas, we can find out whether the performance of the 2021 population is lower than the 2020 population in each of the subject content areas. For example, Figure 9 shows that for booklet 1 and booklet 4 the performance of the 2021 population is lower than that of the 2020 population. Although the size of the drop varies, this drop in performance is seen in almost all subject content areas and booklets (as shown in Appendix B Figure B2). The same holds for the mean scores as shown in Appendix B2 Table B2.

Figure 9: Mean facilities for booklet 1 and booklet 4 for all mathematics subject content areas in 2020 and 2021

Performance on assessment objectives for mathematics

Mathematics items can also be divided by assessment objectives. Figure 10 plots the proportions of total booklet scores achieved for each of the mathematics AOs in 2020 and in 2021. As with the English AOs, this analysis indicates no clear, consistent changes in performance between the AOs from 2020 to 2021. This is supported by the mean facility plots by assessment objectives, which show a similar trend across all assessment objectives and booklets (Appendix B2 Figure B3).

Figure 10: Proportions of mathematics booklet scores by assessment objective for 2020 and 2021

Sub-item level analysis for mathematics

The marking of the mathematics items in the NRT differs from that of English, in that each individual mark (out of the 200 marks in the NRT) is assigned to a specific subject area and assessment objective (AO) and then marked individually as to whether the student has met (1 mark) or not met (0 marks) the specific marking criteria. Each of these marking points is called a sub-item, and this allows us to analyse performance within each subject content area and AO at a more granular level^{[footnote 3]}.

The facility data at the sub-item level is contained in Appendix B2 Table B4. Based on this data we have looked for any evidence of any larger changes in the average performance of sub-items in any particular area of the curriculum (subject content area and AO). There are different ways to look at such changes but the analyses performed were as follows:

Average changes in facility from 2020 to 2021.

Average changes in relative facility from 2020 to 2021. This compares the proportional change in facility, taking into account that subsets of sub-items may tend to be harder or easier depending on the skill involved.

Changes to ranking of facilities from 2020 to 2021. This highlights changes in facility for a subset of sub-items compared with the changes for all other sub-items. The data from this analysis is presented in Table 5.2.

Table 5.2: Sub-item level analysis of mathematics subject content areas.

Mathematics subject content area	Number of sub-items	Average change in facility	Average relative change in facility	Average change in rank order
Algebra	49	-0.031	-0.068	-0.45
Geometry and measures	39	-0.042	-0.125	-2.31
Number	46	-0.037	-0.084	-0.09
Probability and statistics	28	-0.027	-0.071	2.04
Ratio and proportion	38	-0.029	-0.065	1.55
Grand Total	200	-0.034	-0.082	0.00

These three measures produced differing results as they measure changes in performance in different ways. However, the aim is to identify whether any areas of the curriculum had consistent changes in performance across all three measures, suggestive of a meaningful change in item performance relative to the other groups.

It is noticeable that geometry and measures sub-items showed the greatest change by all three measures, suggesting that geometry and measures items may have performed relatively less well in 2021 compared with 2020. A t-test was used to compare the sub-item facilities for this subject content area between 2020 and 2021, as shown in Table 5.3. Although the mean facility was slightly lower in 2021, this difference was not statistically significant (p=0.60). Similar analysis by AO for mathematics produced no clear picture (data not shown).

Table 5.3: Comparison of mean facility for geometry and measures subitems

Geometry and measures subitems	Mean facility	Standard deviation of facility
2020	0.43	2.03
2021	0.38	2.00

Gender

As noted above, this analysis also explored whether boys’ and girls’ performance on the mathematics subject content areas was differentially impacted by the Covid-19 pandemic. Although there have been no significant differences in gender performance in the mathematics NRT, girls tend to do less well than boys. However, with respect to the different curriculum areas, there appear to be no notable differences in the performance of boys and girls across the different curriculum areas between 2020 and 2021. As was the case for English, this finding reflects the fact that across the whole sample very few changes in the performance in the different content areas were observed. Appendix C Figure C2 shows that the proportion of each subject content area contributing to the total score does not change noticeably between the years.

Conclusion for mathematics

The NRT results from 2021 indicate that there was a significant drop in performance in mathematics compared with 2020. This study compares the relative performance by subject content area and assessment objective in mathematics to see whether any relative changes in performance in different areas of the curriculum could be identified that may indicate specific mathematics skills that have proven to be difficult to teach or learn or given different levels of priority as a result of the disruption to the 2021 cohort caused by the Covid-19 pandemic.

The analysis of profile plots, booklet scoring and gender factors shows there were no clear differences in the performance at the booklet level that might indicate any consistent change in curriculum coverage. The inclusion of sub-item data for mathematics provides a method for analysis of potential curriculum effects across the whole of the NRT. Although this analysis suggests that there may be relatively lower performance in marks testing geometry and measures, these differences are not statistically significant.

Comparison of the performance of the different curriculum areas, by item and sub-item analysis, suggests that performance fell for all areas of the curriculum. Therefore, it is reasonable to conclude that disruption due to the Covid-19 pandemic impacted all areas of the mathematics curriculum to a similar degree. As the extent of Covid-related disruption was variable at the school, class and even student level then the absence of a clear pattern in changes in performance at the curriculum level is perhaps unsurprising.

6. References

Bechger, T. and Partchev, I. (2021). How Members of Different Groups Obtain the Same Test Score: Profile Plots in Dexter [online].

7. Glossary of technical terms

7.1 Block interlaced design

In these designs, a booklet contains two blocks of items. The first block of items overlaps with the second block of items of the previous booklet. The second block of items of the last booklet overlaps with the first block of items of the first booklet.

7.2 Booklet

A booklet is the unique collection of items allocated to a test version. Each student will see only one booklet. Booklets are designed to be approximately equivalent in coverage and difficultly so that the experience of each student is comparable.

7.3 Classical Test Theory

Theory that postulates that the observed score is composed of a true score and a measurement error component. CTT measures the difficulty of an item as its facility for the population of students that have completed the item.

7.4 IRT

Item response theory. Theory that specifies the relationship between ability and the probability of answering an item correctly using a mathematical model. IRT measures the difficulty of an item on a theoretical ability scale that is valid for all students in the cohort, not just those that completed that item.

7.5 Item Facility

In Classical Test Theory, simply the proportion of correct answers to an item from the population of students that have completed that item.

7.6 Measurement error

The difference between the reported and true values.

7.7 Standard deviation

A measure of the variation of a survey statistic.

8. Appendix A: Profile plots for all assessment objectives and subject areas

8.1 A1 English profile plots

Figure A1: Reading vs writing

Figure A1 shows eight profile plots for booklets 1 to 8 each showing reading versus writing scores. Each plot maps the scores for reading against the scores for writing, showing trends for the 2020 and 2021 cohorts. There is no evidence of a difference in performance between reading and writing in the NRT between 2020 and 2021.

Figure A2: AO1

Figure A2 shows profile plots for English Booklets 1 to 8 showing AO1 scores. Each plot maps the scores for AO1 against the other assessment objectives, showing trends for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO1 in the NRT between 2020 and 2021.

Figure A3: AO2

Figure A3 shows profile plots for English Booklets 1 to 8 showing AO2 scores. Each plot maps the scores for AO2 against the other assessment objectives, showing trends for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO2 in the NRT between 2020 and 2021.

Figure A4: AO3 (booklets 1,6,7 and 8 contain AO3 items)

Figure A4 shows profile plots for English Booklets 1, 6, 7 and 8 showing AO3 scores. Each plot maps the scores for AO3 against the other assessment objectives, showing trends for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO3 in the NRT between 2020 and 2021.

Figure A5: AO4 (booklets 2,3,4 and 5 contain AO4 items)

Figure A5 shows profile plot for English Booklets 2, 3, 4 and 5 showing AO4 scores. Each plot maps the scores for AO4 against the other assessment objectives, showing trends for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO4 in the NRT between 2020 and 2021.

Figure A6: AO5

Figure A6 shows profile plots for English Booklets 1 to 8 showing AO5 scores. Each plot maps the scores for AO5 against the other assessment objectives, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO5 in the NRT between 2020 and 2021.

Figure A7: AO6

Figure A7 shows profile plots for English Booklets 1 to 8 showing AO6 scores. Each plot maps the scores for AO6 against the other assessment objectives, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance on AO6 in the NRT between 2020 and 2021.

9. A2 Mathematics profile plots

9.1 Figure A8: Subject area – geometry and measures

Figure A8 shows profile plots for Mathematics Booklets 1 to 8 showing geometry scores. Each plot maps the scores for geometry against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance between 2020 and 2021.

9.2 Figure A9: Subject area – number

Figure A9 shows profile plot for Mathematics Booklets 1 to 8 showing number scores. Each plot maps the scores for number against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance between 2020 and 2021.

9.3 Figure A10: Subject area – algebra

Figure A10 shows profile plots for Mathematics Booklets 1 to 8 showing algebra scores. Each plot maps the scores for algebra against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance between 2020 and 2021.

9.4 Figure A11: Subject area – probability

Booklet 7 does not contain any probability items

Figure A11 shows profile plots for Mathematics Booklets 1, 2, 3, 4, 5, 6 and 8 showing probability scores. Each plot maps the scores for probability against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance between 2020 and 2021.

9.5 Figure A12: Subject area – ratio and proportion

Figure A12 shows profile plots for Mathematics Booklets 1 to 8 showing ratio and proportion scores. Each plot maps the scores for ratio and proportion against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear difference in performance between 2020 and 2021.

9.6 Figure A13: Subject area – statistics (booklets 2 and 3 do not contain any statistics items)

Figure A13 shows profile plots for booklets 1, 4, 5, 6, 7 and 8 showing statistics scores. Each plot maps the scores for statistics against the other subject content areas, with trends shown for the 2020 and 2021 cohorts. There is no evidence of a clear change in performance between 2020 and 2021.

10. Appendix B: Proportions, facilities and mean scores

10.1 B1 English

Figure B1: English facilities

Figure B1 comprises charts showing the mean facilities for the English assessment objectives in English Booklets 1 to 8.

Table B1: English mean scores table

Booklet	Assessment objective	Mean score 2020	Mean score 2021	SD score 2020	SD score 2021
1	AO1	2.554	2.541	1.265	1.267
1	AO2	3.826	3.911	1.995	1.987
1	AO3	2.237	2.242	1.750	1.643
1	AO5	6.868	6.675	3.299	3.539
1	AO6	4.621	4.500	2.123	2.293
2	AO1	3.296	3.265	1.134	1.091
2	AO2	2.539	2.585	1.746	1.913
2	AO4	2.218	2.194	1.786	1.796
2	AO5	6.783	6.830	3.214	3.317
2	AO6	4.615	4.611	2.049	2.161
3	AO1	3.380	3.222	1.064	1.064
3	AO2	2.728	2.465	1.801	1.908
3	AO4	2.254	2.214	1.757	1.819
3	AO5	6.936	6.863	3.155	3.333
3	AO6	4.700	4.628	2.026	2.200
4	AO1	2.248	2.262	1.411	1.350
4	AO2	2.893	2.785	2.192	2.250
4	AO4	2.526	2.578	2.081	2.093
4	AO5	6.788	6.639	3.145	3.508
4	AO6	4.660	4.481	2.091	2.289
5	AO1	2.262	2.176	1.348	1.330
5	AO2	2.808	2.772	2.146	2.176
5	AO4	2.565	2.471	2.050	1.941
5	AO5	6.836	6.919	3.309	3.484
5	AO6	4.530	4.646	2.085	2.275
6	AO1	2.379	2.331	1.367	1.417
6	AO2	4.183	3.686	2.171	2.205
6	AO3	2.212	2.170	1.722	1.725
6	AO5	6.909	6.813	3.409	3.691
6	AO6	4.659	4.536	2.192	2.380

10.2 B2 Mathematics

Figure B2: Mathematics – subject area facilities

Chart showing the mean facilities for the maths subject areas in booklet 1. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 2. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 3. This shows a drop in performance is seen in almost all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 4. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 5. This shows a drop in performance is seen in almost all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 6. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 7. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Chart showing the mean facilities for the maths subject areas in booklet 8. This shows a drop in performance is seen in all subject areas. The mean scores and standard deviations are shown in Appendix B2 Table B2.

Table B2: Mathematics – mean scores by subject content areas

Booklet	Subject content area	Mean score 2020	Mean score 2021	SD score 2020	SD score 2021
1	A	3.435	2.782	3.066	2.845
1	G	5.693	4.99	4.008	3.771
1	N	7.302	6.682	3.281	3.607
1	P&S	3.618	3.292	2.813	2.775
1	R	5.422	4.914	2.485	2.665
2	A	4.641	4.259	2.516	2.684
2	G	8.091	7.097	6.066	6.27
2	N	4.621	4.176	2.484	2.493
2	P&S	1.882	1.664	1.92	1.919
2	R	4.993	4.56	3.011	3.112
3	A	5.486	5.201	3.893	4.06
3	G	3.395	2.977	4.066	3.855
3	N	3.979	3.793	2.056	2.151
3	P&S	2.004	1.805	1.413	1.412
3	R	6.961	6.838	3.558	3.699
4	A	5.428	5.304	4.406	4.788
4	G	1.283	1.26	1.698	1.691
4	N	8.105	7.781	4.806	4.876
4	P&S	4.665	4.633	2.66	2.636
4	R	4.203	4.196	1.962	1.961
5	A	7.338	6.935	4.311	4.32
5	G	2.476	2.365	2.181	2.088
5	N	4.91	4.43	3.927	3.746
5	P&S	4.316	4.443	2.297	2.284
5	R	3.088	2.816	2.817	2.779
6	A	7.796	7.342	4.918	4.952
6	G	3.608	3.409	2.723	2.747
6	N	3.559	3.313	1.831	1.923
6	P&S	3.483	3.218	2.008	1.933
6	R	3.072	2.959	2.822	2.826
7	A	4.629	4.115	3.899	3.723
7	G	5.576	4.836	4.265	3.916
7	N	5.749	5.368	2.697	2.814
7	P&S	3.833	3.392	2.425	2.29
7	R	2.404	2.273	1.746	1.791
8	A	3.048	2.686	2.854	2.805
8	G	4.152	3.61	3.495	3.424
8	N	5.907	5.574	2.862	3.013
8	P&S	3.916	3.662	2.249	2.161
8	R	5.886	5.587	2.917	3.069

Figure B3: Mathematics – assessment objectives facilities

Chart showing the mean facilities for the maths assessment objectives in booklet 1. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 2. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 3. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 4. This shows a drop in performance is seen in almost all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 5. This shows a drop in performance is seen in almost all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 6. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 7. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Chart showing the mean facilities for the maths assessment objectives in booklet 8. This shows a drop in performance is seen in all assessment objectives. The mean scores and standard deviations are shown in Appendix B2 Table B3.

Table B3: Mathematics mean scores by assessment objective

Booklet	Assessment objective	Mean score 2020	Mean score 2021	SD score 2020	SD score 2021
1	1	10.188	9.109	5.697	5.599
1	2	7.186	6.515	3.703	3.764
1	3	8.096	7.037	4.551	4.462
2	1	9.685	8.730	6.491	6.528
2	2	7.054	6.214	3.633	3.776
2	3	7.489	6.811	4.446	4.620
3	1	7.558	7.214	5.894	5.844
3	2	9.790	9.105	3.953	4.123
3	3	4.477	4.294	3.962	3.882
4	1	7.495	7.473	5.019	5.245
4	2	10.861	10.521	4.905	4.977
4	3	5.328	5.179	4.390	4.551
5	1	9.011	8.568	5.232	5.112
5	2	6.014	6.023	3.292	3.263
5	3	7.102	6.397	5.852	5.722
6	1	11.480	10.809	5.690	5.837
6	2	5.439	5.010	4.132	4.073
6	3	4.598	4.422	3.089	3.150
7	1	11.799	10.772	6.456	6.344
7	2	7.060	6.195	5.319	4.992
7	3	3.333	3.018	1.910	1.867
8	1	10.438	9.565	5.542	5.617
8	2	6.761	6.382	3.234	3.244
8	3	5.711	5.173	4.115	4.113

Table B4: Mathematics sub-item facilities 2020 and 2021

Subitem ID	Assessment objective	subject content area	Facility 2020	Facility 2021
M102_202_X1	2	P&S	0.73926	0.680892
M102_202_X2	1	R	0.720167	0.653734
M102_202_X3	2	R	0.585322	0.519884
M103_203_X1	3	N	0.838305	0.777886
M103_203_X2	1	N	0.78222	0.737148
M103_203_X3	3	N	0.841885	0.786615
M103_203_X4	1	N	0.671838	0.631426
M103_203_X5	3	N	0.597852	0.553831
M105_204_X1	3	G	0.705251	0.627546
M105_204_X2	1	N	0.668258	0.599418
M105_204_X3	1	G	0.634845	0.561591
M106_207_X1	1	A	0.72494	0.686712
M106_207_X2	1	A	0.541169	0.479146
M108_208_A1_1	2	G	0.654535	0.609117
M108_208_A2_1	1	G	0.529236	0.493695
M108_208_B1	2	G	0.683174	0.621726
M108_208_B2	2	G	0.443914	0.375364
M108_208_B3	2	G	0.383652	0.304559
M109_209_A1	2	P&S	0.560263	0.483996
M109_209_A2	2	P&S	0.415274	0.380213
M109_209_B1	3	P&S	0.424821	0.373424
M109_209_B2	1	P&S	0.279236	0.234724
M109_209_B3	1	P&S	0.220167	0.191077
M113_213_X1	1	A	0.2142	0.183317
M113_213_X2	1	A	0.168854	0.13967
M201_303_A1	2	A	0.74477	0.670223
M201_303_B1	2	A	0.642558	0.57808
M201_303_B2	1	R	0.466228	0.42192
M201_303_C1	2	A	0.80275	0.738118
M201_303_C2	2	A	0.390915	0.336566
M205_305_X1	3	R	0.635983	0.609117
M205_305_X2	3	R	0.579797	0.553831
M205_305_X3	3	R	0.631201	0.617847
M205_305_X4	1	N	0.53676	0.513094
M205_305_X5	3	N	0.477585	0.451018
M206_306_X1	1	G	0.368201	0.320078
M206_306_X2	1	G	0.306037	0.276431
M206_306_X3	1	G	0.176928	0.162949
M210_309_X1	1	G	0.391512	0.329777
M210_309_X2	3	G	0.364017	0.293889
M210_309_X3	3	G	0.225941	0.204656
M210_309_X4	3	N	0.205619	0.187197
M211_310_X1	3	G	0.454274	0.42192
M211_310_X2	1	G	0.298267	0.260912
M211_310_X3	3	R	0.240885	0.225024
M211_310_X4	3	R	0.175732	0.14743
M211_310_X5	1	N	0.17633	0.153249
M212_312_X1	1	N	0.297669	0.263822
M212_312_X2	1	N	0.273162	0.231814
M212_312_X3	1	N	0.256426	0.215325
M301_401_X1	1	R	0.83597	0.826718
M301_401_X2	1	R	0.776801	0.755082
M302_402_A1	2	N	0.883421	0.845111
M302_402_A2	2	N	0.847686	0.812197
M302_402_B1	2	N	0.730521	0.703775
M302_402_B2	2	N	0.671353	0.653437
M304_404_X1_V2	2	P&S	0.662566	0.659245
M304_404_X2_V2	2	R	0.865261	0.835431
M304_404_X3_V2	1	R	0.558875	0.569216
M304_404_X4_V2	2	R	0.496778	0.485963
M307_407_X1	1	A	0.605155	0.594385
M307_407_X2	1	A	0.564148	0.550823
M308_408_A1_1	2	P&S	0.678969	0.667957
M308_408_A2_1	2	P&S	0.492091	0.435624
M308_408_B1	2	P&S	0.482718	0.451113
M308_408_B2	2	P&S	0.346807	0.324298
M311_411_X1	3	G	0.362624	0.352372
M311_411_X2	3	A	0.144698	0.164569
M311_411_X3	1	A	0.15993	0.181994
M311_411_X4	1	A	0.116579	0.125847
M311_411_X5	3	A	0.070885	0.083253
M313_413_X1	1	A	0.173989	0.17909
M313_413_X2	1	A	0.12478	0.135528
M313_413_X3	1	A	0.126538	0.135528
M313_413_X4	1	A	0.037493	0.040658
M403_502_X1	2	P&S	0.776801	0.780115
M403_502_X2	2	P&S	0.592853	0.602294
M403_502_X3	1	P&S	0.703574	0.713193
M403_502_X4	2	P&S	0.545987	0.556405
M405_504_A1	3	A	0.644405	0.59847
M405_504_A2	3	A	0.608084	0.558317
M405_504_A3	1	A	0.600469	0.555449
M405_504_B1	3	A	0.54833	0.478967
M405_504_B2	1	R	0.420621	0.382409
M406_506_A1	2	N	0.695372	0.669216
M406_506_A2	2	N	0.552431	0.541109
M406_506_B1	2	N	0.508494	0.384321
M406_506_C1	3	N	0.668424	0.64914
M409_509_X1	3	G	0.334505	0.328872
M409_509_X2	3	G	0.356766	0.347992
M409_509_X3	1	G	0.344464	0.31262
M409_509_X4	3	R	0.222613	0.195985
M410_510_X1	3	N	0.447569	0.422562
M410_510_X2	1	R	0.37727	0.358509
M410_510_X3	3	N	0.308143	0.285851
M410_510_X4	3	N	0.314001	0.291587
M410_510_X5	3	N	0.20621	0.185468
M412_512_A1	1	N	0.436438	0.402486
M412_512_B1	1	N	0.250146	0.222753
M412_512_B2	1	N	0.203866	0.183556
M501_601_A1	1	A	0.896429	0.877885
M501_601_B1	1	A	0.8	0.753846
M501_601_B2	1	A	0.6625	0.623077
M503_602_A1	2	P&S	0.882143	0.874038
M503_602_B1	2	P&S	0.497024	0.516346
M503_602_B2	2	P&S	0.391071	0.438462
M505_604_A1	1	A	0.578571	0.578846
M505_604_B1	1	A	0.635119	0.568269
M507_607_X1	1	G	0.673214	0.664423
M507_607_X2	1	N	0.567262	0.565385
M508_609_A1	3	R	0.5625	0.552885
M508_609_A2	3	R	0.45119	0.4375
M508_609_B1	3	R	0.31131	0.296154
M508_609_B2	3	R	0.178571	0.166346
M508_609_B3	3	R	0.155357	0.150962
M511_610_A1	3	R	0.335714	0.286538
M511_610_A2	3	R	0.3	0.252885
M511_610_A3	1	R	0.271429	0.232692
M511_610_B1	2	R	0.51369	0.510577
M513_613_A1	1	A	0.578571	0.597115
M513_613_B1_1	2	G	0.085119	0.095192
M513_613_B1_2	2	A	0.079167	0.082692
M513_613_B2_1	3	A	0.155952	0.146154
M513_613_B2_2	1	A	0.143452	0.133654
M513_613_B2_3	3	A	0.025595	0.024038
M603_704_X1	3	N	0.889741	0.84518
M603_704_X2	1	N	0.748821	0.687439
M603_704_X3	3	N	0.706958	0.67186
M603_704_X4	1	R	0.650354	0.599805
M603_704_X5	3	N	0.568396	0.519961
M605_706_A1	1	A	0.599646	0.576436
M605_706_A2	1	A	0.489387	0.461538
M605_706_B1	2	A	0.455778	0.402142
M605_706_B2	2	A	0.313679	0.280428
M605_706_B3	1	A	0.299528	0.264849
M606_708_A1	1	P&S	0.799528	0.743914
M606_708_B1	2	P&S	0.335495	0.259007
M606_708_B2	2	P&S	0.310731	0.238559
M606_708_B3	2	P&S	0.247052	0.178189
M608_710_A1_1	2	A	0.407429	0.360273
M608_710_A2_1	2	A	0.237618	0.208374
M608_710_B1	1	A	0.222877	0.164557
M608_710_B2	2	A	0.129127	0.079844
M611_712_A1	1	G	0.59316	0.552093
M611_712_A2	1	G	0.464623	0.428432
M611_712_B1	2	G	0.341392	0.308666
M611_712_C1	2	G	0.173349	0.140214
M611_712_C2	2	G	0.098467	0.070107
M612_713_A1	1	G	0.440448	0.397274
M612_713_B1	1	G	0.252358	0.232717
M701_802_X1	1	R	0.726521	0.697786
M701_802_X2	1	R	0.144123	0.14437
M702_803_A1	2	P&S	0.969876	0.965351
M702_803_B1	2	P&S	0.412286	0.368624
M702_803_B2	1	R	0.395157	0.3436
M702_803_B3	2	R	0.369167	0.322425
M703_805_A1	1	G	0.565859	0.524543
M703_805_B1	1	G	0.484938	0.384986
M703_805_C1	1	G	0.645009	0.59384
M705_807_A1	1	N	0.853514	0.828681
M705_807_B1	2	N	0.464265	0.430221
M705_807_C1	2	N	0.498523	0.413859
M705_807_C2	2	N	0.368576	0.392685
M707_808_A1	1	G	0.536917	0.481232
M707_808_B1	1	G	0.536326	0.512993
M709_809_A1	1	A	0.594211	0.546679
M709_809_B1	2	A	0.531601	0.482194
M709_809_B2	2	A	0.319551	0.274302
M711_810_X1	3	R	0.6456	0.606352
M711_810_X2	1	R	0.590667	0.536092
M711_810_X3	3	R	0.21264	0.194418
M711_810_X4	1	R	0.13231	0.133782
M714_814_X1	3	G	0.168931	0.115496
M714_814_X2	1	G	0.165978	0.110683
M714_814_X3	3	G	0.134672	0.087584
M801_101_X1	1	P&S	0.62381	0.590953
M804_104_A1	2	R	0.844643	0.806545
M804_104_A2	2	N	0.532143	0.471607
M804_104_B1	3	R	0.790476	0.73051
M804_104_B2	3	R	0.74881	0.694899
M804_104_B3	1	R	0.463095	0.438884
M806_107_A1	2	N	0.833929	0.790183
M806_107_A2	1	N	0.783929	0.733397
M806_107_A3	2	N	0.559524	0.512993
M806_107_B1	3	N	0.442262	0.39461
M806_107_B2	3	N	0.717857	0.655438
M806_107_B3	1	N	0.317857	0.274302
M811_110_A1	1	A	0.203571	0.171319
M811_110_B1	3	A	0.417857	0.350337
M811_110_B2	3	A	0.36369	0.298364
M811_110_B3	1	A	0.348214	0.289702
M811_110_B4	3	A	0.32381	0.268527
M812_111_X1	3	G	0.285714	0.231954
M812_111_X2	3	G	0.23631	0.176131
M812_111_X3	3	G	0.179762	0.142445
M812_111_X4	1	N	0.125595	0.095284
M812_111_X5	3	N	0.122024	0.093359
M813_112_A1	1	P&S	0.61369	0.570741
M813_112_B1	1	P&S	0.48869	0.446583
M813_112_C1	2	P&S	0.004167	0.00385

11. Appendix C: Gender and proportion of booklet score

11.1 Figure C1: English

Figure C1 comprises charts showing the proportions of booklet 1 to 8 scores by English language assessment objectives for boys and girls in 2020 and 2021. There appear to be no notable changes in the performance of boys and girls across the different assessment objectives (AOs) between 2020 and 2021.

11.2 Figure C2: Mathematics

Figure C2 comprises charts showing the proportions of booklet 1 to 8 scores by maths subject area for boys and girls in 2020 and 2021. There appear to be no notable differences in the performance of boys and girls across the different curriculum areas between 2020 and 2021.

---

12. Document information

12.1 Authors

• Bethan Burge
• Louise Benson
• Alex Cutbill
• Sarah Gibb
• David Thomas
• Jasper Wouda

12.2 Citation

How to cite this publication:

Burge, B., Benson, L., Cutbill, A., Gibb, S., Thomas, D. and Wouda, J. (2022) National Reference Test 2021: Performance Across Content Areas. Slough: NFER

12.3 Copyright

© National Foundation for Educational Research 2022

All rights reserved. No part of this document may be reproduced or transmitted

in any form or by any means, electronic, mechanical, photocopying, or otherwise,

without prior written permission of NFER.

NFER, The Mere, Upton Park, Slough, Berks SL1 2DQ
Telephone +44 (0)1753 574123
Fax +44 (0)1753 691632

Find out about call charges

Email enquiries@nfer.ac.uk
NFER website

NFER reference: OFMT

ISBN: 978-1-912596-70-6

---

1. NRT 2020 was held in February 2020, before the Covid-related closure of schools in late March 2020. ↩

2. NRT 2021 was held in April 2021, after schools returned following school closures in spring 2021. ↩

3. A five-mark item may contain more than one subject content area and more than one assessment objective, thereby merging performance over these different areas or skills. Analysis at the sub-item level, in theory removes this merging effect so analysis will look at performance solely on the specific subject content area or assessment objective skill. ↩