Official Statistics

Armed forces continuous working patterns survey 2023/24: background quality report

Published 25 July 2024

1. Contact

The Responsible Statistician for the Armed Forces Continuous Working Patterns (CWP) is the Head of the Analysis Directorate Surveys Team. Email: Analysis-Surveys-Enquiries@mod.gov.uk

2. Introduction

The Continuous Working Patterns (CWP) Survey is a seven-day diary completed by trained, UK regular personnel to record the number of hours spent at work, on call, on breaks and off duty. The CWP report is produced annually and contains information about the working patterns of trained, UK regular personnel over the last seven years, with comparisons made between the current survey and the previous surveys.

The Chief of Defence People (CDP) sponsors the survey under a remit from the Armed Forces Pay Review Body (AFPRB), which provides advice to the Prime Minister and the Secretary of State for Defence on the remuneration and charges for members of the UK Armed Forces.

Analysis Surveys have been providing the AFPRB with information about the working patterns of trained, regular UK Armed Forces personnel since 1987/88. Since 1987/88, a number of changes have been made to the survey. For example, prior to 1995/96 the survey excluded Officers, and prior to 1996/97, Royal Marines were excluded. The locations for the Army and RAF have changed over the years depending on where trained, UK regular personnel have been stationed or deployed. The analysis methodology has also changed, moving from substitution to calculating a notional week, based upon the average Monday, the average Tuesday, …., the average Sunday.

3. Statistical Processing

Stage 1: Questionnaire design

Single Service questionnaires are used for the Royal Navy (Royal Navy and Royal Marines), the Army and RAF. All three single Service questionnaires contain a seven-day diary, with each day divided into four periods of six hours. Respondents are asked to record the number of hours they spend at work, on breaks, on call and off duty for all four periods of each day. Guidance on how different activities should be recorded are provided.

CWP 2023/24 questionnaires are published as a separate document and can be found on the CWP webpage.

Stage 2: Sample

The target population was regular members of the UK Armed Forces who were full time, trained strength. It excluded untrained personnel, those on long term absence, Special Forces, Gurkhas, reservists and personnel ranked above OF6. There were a number of other minor exclusions arising from the practicalities of running the survey e.g. those with invalid address data. Address data for personnel in the sample were obtained from the Joint Personnel Administration System (JPA).

CWP is designed as a stratified sample survey. Stratification is by:

1. Service: Royal Navy, Army and RAF.
2. Rank group: Senior Officers (OF3 to OF6), Junior Officers (OF1 to OF2), Senior Ranks/Rates (OR6 to OR9) and Junior Ranks/Rates (OR1 to OR4).
3. Broad location:
   • Royal Navy: sea and shore.
   • Army: United Kingdom, Germany, overseas Operations and elsewhere abroad.
   • RAF: United Kingdom, overseas Operations and elsewhere abroad.

Stage 3: Distribution

In previous years CWP was conducted using only paper surveys. In the later stage of the data collection process (May 2021) of CWP 2020/21, an equivalent online version of the survey was introduced. For CWP 2023/24, both paper and online surveys are distributed to personnel. Paper questionnaires are packed by an external contractor and dispatched via the British Forces Post Office, RAF Northolt. Lime survey is the platform used for distributing the online survey. Questionnaires are sent out in two waves, the first in October 2023 and the second in January 2024, allowing responses to be collected between October 2023 and March 2024.

Since 2018/19, when a unit or location had a large enough sample, the questionnaires are batched and when possible sent to a named point of contact. Prior to 2018/19, questionnaires were distributed in batches labelled for the Commanding Officer.

The CWP Survey 2023/24 was distributed to 26,472 personnel. A response is considered valid if at least one day within the seven-day diary is useable. 3,346 of the returned surveys were considered valid, which equated to a response rate of 13%. Of these valid surveys, 2,862 responses were online survey returns. The response rate in 2023/24 (13%) has remained similar to the previous year (14%).

Stage 4: Analysis

In order to detect any statistical differences in working patterns between the current year and the previous year, a series of z-tests were conducted with an alpha level of 5%. A statistically significant difference means there is a less than 5% probability that the difference is the result of chance alone. If a statistical difference is found, it means that the difference between years is unlikely to be the result of random variation and is therefore indicative of a genuine change in hours spent at work, on duty or on call. It does not mean that the change is necessarily large or substantively “important”. It is important to note that the absence of a statistically significant difference between years does not necessarily mean that no difference is expected to exist between populations. Simply that, given the number of respondents, the detected difference is too small for us to be confident that a difference of this size could not have arisen due to chance variation in the survey process.

4. Quality Management

4.1 Quality Assurance

The MOD’s quality management process for Official Statistics consists of three elements: (1) Regularly monitoring and assessing quality risk via an annual assessment; (2) Providing a mechanism for reporting and reviewing revisions/corrections to Official Statistics; (3) Ensuring Background quality reports are published alongside reports and are updated regularly.

4.2 Quality Assessment

The most recent internal quality assessment of CWP was carried out in November 2023. The quality risk of CWP was assessed as low. The main risks to quality relate to response rates and documentation and are discussed further in the accuracy and reliability section of this BQR.

5. Relevance

The principal users of CWP are the Chief of Defence People (CDP) policy team, who sponsor the survey, and the Armed Forces Pay Review Body (AFPRB). The AFPRB uses the information about the working patterns of Service personnel in CWP when providing advice to the Prime Minister and the Secretary of State for Defence on the remuneration and charges for members of the UK Armed Forces.

The statistics and information in the CWP report can be used to answer parliamentary questions and Freedom of Information requests; they can also be used by the general public and media to monitor the working hours of trained, regular Armed Forces personnel.

6. Accuracy and Reliability

6.1 Overall Accuracy

CWP collects data about working patterns of trained, UK regular personnel from a stratified sample of approximately 26,472 personnel. The sample is stratified by Service (Royal Navy (Royal Navy and Royal Marines), Army and RAF), rank group and broad location.

The survey is designed to achieve a margin of error within +/-0.8 hours for each single Service average hours worked estimate and +/- 1.3 hours for each rank group average hours worked estimate.

Table 1 shows a wider margin of error for the Royal Navy Senior Ranks/Rates estimate of average hours worked.

Table 1: Margins of error for average hours worked 2023/24 by rank group

	UK Armed Forces Margin of error (in hours)	Royal Navy Force Margin of error (in hours)	Army Force Margin of error (in hours)	RAF Force Margin of error (in hours)
All Ranks	0.45	0.76	0.72	0.56
Senior Officers	0.53	1.28	0.74	0.79
Junior Officers	0.55	1.19	0.86	0.87
Senior Ranks/Rates	0.59	1.41	0.87	0.70
Junior Ranks/Rates	0.74	1.22	1.13	1.01

Table 2 shows estimates of average hours worked by locations typically have higher margins of error due to limited population sizes and low response rates.

Table 2: Margins of error for average hours worked 2023/24 by broad location

Service	Broad location	Margin of error (in hours)
Royal Navy	Ship/Sea	1.84
	Shore	0.82
Army	United Kingdom	0.78
	Overseas Operations	1.89
	Elsewhere abroad	1.42
RAF	United Kingdom	0.61
	Overseas Operation	1.81
	Elsewhere abroad	1.05

For all three Services, rank groups and broad locations, levels of precision were lower (wider confidence intervals) for on duty and on call averages.

6.2 Sampling/Non-sampling errors

Many respondents returned questionnaires which included leave days or days that had to be discarded because of inconsistent or missing data. If analysis was restricted to only those questionnaires that cover a full working week, results would be based on much less data and confidence intervals would be considerably wider. Therefore, the methodology used is based upon a ‘notional’ week made up of the average Monday, the average Tuesday, …., the average Sunday. By calculating the average working hours separately for each day, as much of the data as possible is used.

The sample design and the difference in non-response between single Services, rank groups and broad locations mean that the distribution of characteristics of respondents do not reflect their distribution in the trained, regular, UK Armed Forces. This means that some types of personnel were over-represented and others underrepresented in the data. The survey data are weighted by single Service, rank group and broad location to ensure that the respondent profile matched the known population profile by these characteristics. The weights are calculated simply by dividing the population size within each weighting class (p) by the number of responses within the same weighting class (r). Weighting in this way assumes missing data are missing at random (MAR) only within strata. This means we assume that within a single strata, the working patterns of non-respondents are similar (on average) to the working patterns of respondents. If trained, regular UK Armed Forces personnel who did not respond to the CWP survey have different working patterns to those who did, than the CWP survey results will be biased.

Non-response that is directly related to individual working patterns will lead to bias within the survey results. For example, those busiest and hence working longer hours may be less likely to complete the survey.

6.3 Data Revisions

Data revisions are handled in accordance with the MOD’s Official Statistics Revisions and Corrections Policy.

Figures may have been revised in previous reports and so may not match previously published figures. Revised estimates are identified by a ‘r’ marker, with an accompanying footnote to explain the change.

There are no scheduled revisions to CWP. Any required corrections in addition to the above will be released in updated reports and/or reference tables, along with the reasons for the corrections, on the GOV.UK website.

6.4 Impact of COVID-19

CWP survey 2020/21 was in field from November 2020 to June 2021, during a time of national restrictions imposed as a result of the COVID-19 pandemic, and two national lockdowns which may have influenced results. The potential impact of COVID-19 on personnel’s working pattern should still be considered when interpreting the results presented for previous years. Some working patterns reported in 2022/23, when compared to working patterns reported in 2020/21, have reverted back to levels reported in 2019/20. COVID-19 was present for a smaller proportion of time in this period and may have impacted working patterns less.

To assess the potential impact of COVID-19 on working patterns of personnel, a question has been added in the survey since 2020/21: “Were your working patterns for this week affected by the current COVID-19 situation?” Of the respondents to the questions, in 2020/21, just over a third (36%) reported that their working patterns were affected by COVID-19.

Since 2022/23, due to a low response rate to the additional COVID-19 questions, there has been insufficient data available to complete the analysis. Consequently, any analysis that was completed on the data produced mixed results and failed to provide meaningful conclusions. Thus, the decision was made to exclude any COVID-19 analysis from additional tables (section 9) in the accompanying reference tables; however, comparison from previous years can still be viewed here.

6.5 Introduction of online survey

An online version of the CWP survey was introduced towards the end of CWP 2020/21 data collection process. This year marked the third year of full introduction of online survey to CWP. Whilst the online survey allows us to reach more of our sampled personnel than using paper survey solely, an additional mode of survey instrument might have an impact on the responses we captured. Consequently, results from paper survey and online survey were analysed both separately and as a combined dataset. The main report focuses on the combined results.

6.6 Personnel deployed on overseas operations

Although we aim to sample all personnel deployed on overseas operations, the reduction in the number of personnel deployed and the difficulties making contact with those who are, has led to a reduction in the number of responses from deployed personnel in recent years. In 2019/20 and 2020/21, we trialled sending packs of unnamed questionnaires to points of contact in overseas Operational locations. The points of contact arranged distribution of questionnaires to all personnel at their location. Since 2021/22, the trial was also carried out by online survey. Since 2022/23, to capture as many personnel deployed on overseas operations as possible, a link for the online survey was provided to point of contacts based in overseas operational locations. Contacts were asked to circulate the link among all personnel based in their deployed locations and encourage completion of the survey. Response rates are recorded and presented in the reference table to monitor trial results.

To better understand the possible effect of an additional survey for deployed personnel on the overall results for personnel deployed, responses from the deployed surveys trial were analysed separately and compared to the analysis of the original deployed sample. The main report and accompanying reference tables focus on combined responses from the deployed surveys trial and the original deployed sample surveys.

6.7 Personnel based in Germany

In line with the announcement of the Strategic Defence and Security Review (SDSR) of 2010, the Army Basing Programme (ABP) was established to rebase the British Army from Germany to the UK by 2020. Large numbers of personnel moved from Germany back to the UK over the summer of 2019. This led to very low levels of response from Germany. Breakdowns for Germany have not been included in this report since 2019/20. However, Germany responses are still included in the overall Army and Rank group averages.

Sampling methods are under review to better batch surveys. The Surveys Sampling tool (STATS) is in the process of being updated to incorporate these changes.

6.8 Documentation

Detailed documentation of the processing is available and is currently being collated into one document. Documentation on data analysis is available and is relatively easy to follow for someone who has never been involved with CWP. This year we are trying to capture changes to the process in one document. This currently needs to be reviewed.

With continuous improvements on processes, the introduction of online survey and the trial of deployed surveys, document sections on methodology to select and batch samples as well as launching for online survey and deployed surveys trial requires continuous updates.

7. Timeliness and Punctuality

7.1 Timeline

To deconflict with other survey launch dates, CWP 2023/24 timeline is similar to 2022/23, with data collection running from October 2023 to March 2024 in 2023/24. We have shortened the data collection period over the last few years as a staggered approach to bring the publication date forward to late July over the last 2 annual survey cycles. The statistics reported in 2023/24 represent working patterns during the data collection period.

Once the data has been collected, it took approximately four months this year for Analysis Surveys staff to produce and and publish the statistical report.

7.2 Punctuality

As an Official Statistic, the release date for the CWP report was pre-announced on the GOV.UK statistical release calendar section of GOV.UK in accordance with the guidance set out in the Code of Practice for Official Statistics. All preannounced publication deadlines have been met.

8. Coherence and Comparability

8.1 Coherence

In February 2014, the Army’s Family Federation produced a Working Hours Survey. However, neither the content nor the methodology is harmonised with CWP. Information about the working patterns of Army personnel were obtained by asking families of Service personnel multiple choice questions such as ‘in the last two years, have your soldier’s working hours increased?’ and ‘on average, how many hours a day does your soldier work for?’. The Working Hours Survey was conducted using online questionnaires and face to face interviews. Otherwise, CWP is the only survey directly measuring working patterns of Armed Forces personnel.

8.2 Comparability over Time

CWP surveys are considered to be comparable over time. However, there have been changes to the location categories for the Army and RAF to reflect where Army and RAF personnel are stationed or deployed. Where possible, revisions have been made to figures from previous surveys to ensure comparability over time. The online survey was set up to match the paper survey closely to minimise effect on comparability of responses collected from different survey instruments. Additional questions and analyses have been conducted to understand impact of COVID-19 on responses and differences in responses in online and paper survey, whilst maintaining comparability; however, since 2022/23 the analyses was not reported due to insufficient data obtained. Please refer to section 6.4, 6.5 and 6.6 for further information.

9. Accessibility and Clarity

9.1 Accessibility

The CWP report this year is published on the Statistics at MOD area of the GOV.UK website as an accessible HTML web-page. Alongside this report, this Background Quality Report is also published as an accessible HTML web-page whilst the background tables are available in Excel and ODS format. Previous years of the main CWP report and the Background Quality Report have been published as accessible PDF documents.

9.2 Clarity

In addition to this background quality report, the start of the CWP report contains a key points and trends section that summarises the main CWP findings and an introduction that provides information about response rates and the statistical tests used. After the results sections, there is a methodology section that contains more detailed information about the target population, the questionnaire, weighting, cleaning and analysing the data. A glossary is also provided which defines the terms used throughout the report.

Tables, graphs and commentary are used to illustrate trends in the working patterns of trained, regular, UK Armed Forces personnel over the past eight years, and to highlight significant differences between the 2022/23 and the 2023/24 surveys. Where there have been issues with the data or time series comparisons, or revisions made, this has been noted in the relevant section of the report.

10. Trade-offs between Output Quality Components

The main trade-off is between comparability over time, cost and accuracy. From 2012/13, CWP questionnaires moved from being issued in three or four waves to being issued in two waves – this was to lower the administration costs. The rationale for distributing questionnaires in several waves during the year was to obtain survey estimates that were more representative of the entire year and less influenced by seasonal variations, events or announcements. If responses do vary systematically (depending on the period of data collection) the change in number of waves may impact the accuracy of our estimates and comparability of responses between surveys.

11. Cost and Respondent Burden

Costs are closely monitored, and the Surveys team and the working group strive to balance quality and timeliness against costs. The sample size is calculated to be the most efficient in order to meet the levels of precision outlined in Section 6.

Response to CWP is voluntary. Participant information is provided within the questionnaire to encourage informed consent. The CWP questionnaire is currently estimated to take between ten and thirty minutes to complete every day. Therefore, this survey could take between one hour and ten minutes and three hours and thirty minutes to complete over the seven-day period.

12. Confidentiality and Security

12.1 Confidentiality – Policy

Confidentiality protocols are adhered to, as set out in the Defence Statistics Disclosure and Confidentiality Policy – Identifiable Survey Data. CWP is a confidential survey rather than anonymous. The raw data is not seen by anyone outside of Analysis Surveys which ensures that no person from any respondents’ chain of command is able to access individual level data. Only aggregated results are provided to anyone not directly involved with the analysis. Results where the responding group size is less than 30 are not presented as results for groups of this size considered too unreliable and may be disclosive. Each questionnaire is distributed with a participant information sheet to ensure that respondents make an informed decision before completing the survey. The CWP report is published with details of the methodology and a background quality report so that users are aware of potential data issues and quality of the output.

A list of those that require 24 hour pre-release access to the statistical results are published on the MOD statistics pre-release access list section of the GOV.UK website.

12.2 Security

All staff involved in the CWP production process adhere to the MOD and Civil Service data protection regulations. In addition, all members of the survey analysis team have to follow the relevant codes of practice for the Government Statistical Services (GSS). All data is stored, accessed and analysed using the MOD’s restricted network and IT system.

Last updated: July 2024