Data quality and methodology (TSM 2024)
Tenant Satisfaction Measures - Official statistics in development 2023-2024
Applies to England
Documents
Details
Contents
Introduction
This report describes the quality assurance arrangements for the registered provider (RP) Tenant Satisfaction Measures statistics, providing more detail on the regulatory and operational context for data collections which feed these statistics and the safeguards that aim to maximise data quality.
Background
The statistics we publish are based on data collected directly from local authority registered provider (LARPs) and from private registered providers (PRPs) through the Tenant Satisfaction Measures (TSM) return. We use the data collected through these returns extensively as a source of administrative data. The United Kingdom Statistics Authority (UKSA) encourages public bodies to use administrative data for statistical purposes and, as such, we publish these data.
These data are first being published in 2024, following the first collection and publication of the TSM.
Official Statistics in development status
In February 2018, the UKSA published the Code of Practice for Statistics. This sets standards for organisations producing and publishing statistics, ensuring quality, trustworthiness and value.
These statistics are drawn from our TSM data collection and are being published for the first time in 2024 as official statistics in development.
Official statistics in development are official statistics that are undergoing development. Over the next year we will review these statistics and consider areas for improvement to guidance, validations, data processing and analysis. We will also seek user feedback with a view to improving these statistics to meet user needs and to explore issues of data quality and consistency.
Change of designation name
Until September 2023, ‘official statistics in development’ were called ‘experimental statistics’. Further information can be found on the Office for Statistics Regulation website.
User feedback
We are keen to increase the understanding of the data, including the accuracy and reliability, and the value to users. Please complete the form or email feedback, including suggestions for improvements or queries as to the source data or processing to enquiries@rsh.gov.uk.
Publication schedule
We intend to publish these statistics in Autumn each year, with the data pre-announced in the release calendar.
All data and additional information (including a list of individuals (if any) with 24 hour pre-release access) are published on our statistics pages.
Quality assurance of administrative data
The data used in the production of these statistics are classed as administrative data. In 2015 the UKSA published a regulatory standard for the quality assurance of administrative data. As part of our compliance to the Code of Practice, and in the context of other statistics published by the UK Government and its agencies, we have determined that the statistics drawn from the TSMs are likely to be categorised as low-quality risk – medium public interest (with a requirement for basic/enhanced assurance).
The publication of these statistics can be considered as medium public interest, as there has been mainstream media interest, but they have only moderate economic and/or political sensitivity. Concerns over data quality are in general considered low given the data checks by providers and ourselves and subsequent reviews conducted on the submitted data and analytical processes. We have provided more information on data checks and the outcomes of our assessment for individual landlords below.
Notwithstanding this, we aim for the highest standards of data quality possible within the constraints of available resources and the existing regulatory and operational context. Through ongoing internal analysis, we seek to understand the strengths and limitations of the data, the overall quality of the data and to identify potential means by which it may be improved.
Regulatory context
The regulatory framework for social housing in England underpins the collection of TSM data. For more information about the Regulator of Social Housing (RSH) and the regulatory framework please see our website.
Regulatory standards
We collect TSMs data in order to support our operational approach to regulating the consumer standards set out in the regulatory framework for social housing in England.
Landlords are required to collect, publish and report TSMs. The Tenant Satisfaction Measures Standard covered the period 1 April 2023 to 31 March 2024, with the required outcomes and specific expectations included in the Transparency, Influence and Accountability Standard from 1 April 2024. This includes requirements relating to the generation and calculation of individual TSMs that landlords must follow.
Landlords that own 1,000 or more social housing units are also required to submit their TSM results and further contextual information to us. footnote 1
The individual submissions from landlords depend on their holdings of low cost rental accommodation (LCRA) and low cost home ownership accommodation (LCHO) footnote 2. In general, landlords are required to submit responses for the tenant perception and complaints TSMs for a stock type where they own 1,000 or more units. This is explained further in the TSM requirements documents. All landlords owning 1,000 or more units must report the management information TSMs on building safety and anti-social behaviour based on all LCRA and all LCHO stock combined.
A very small number of landlords own fewer than 1,000 units of low cost rental accommodation and fewer than 1,000 units of low cost home ownership accommodation, whilst in total owning 1,000 or more units of social housing. These landlords are permitted to submit combined LCRA and LCHO results covering tenant perception and complaints. Two landlords have submitted in this way. Given the small number of submissions analysis of these two combined submissions would lack robustness. We have not included these landlords in our analysis alongside the breakdown of LCRA and LCHO for the tenant perception and complaint TSM areas.
The regulatory framework and data quality
In Regulating the Standards, we set out the operational approach to regulation and emphasise the importance of providers supplying timely and accurate data.
Governance of data and statistics at RSH
The team responsible for the publication of these statistics is also involved in the TSM data collection and the cleansing of incoming data, and works to support the review of data by regulatory engagement colleagues.
All incoming data are stored and analysed within password-protected government secure networks and access to the sector level analysis work undertaken on the data is restricted until after publication (RP level data is accessed by our staff as part of operational work). Further information on the data quality assurance processes we employ is provided in the TSM collection section.
TSM review
We are reviewing the TSM submissions of all large landlords. As part of that review, we identify landlords which appear to be outliers based upon the TSM data they have supplied. Where this is the case, we may ask them for extra information, either
- because we have concerns about the quality of their TSM data, or
- because the landlord has been identified as being potentially at a higher risk of failing to deliver the outcomes of our standards.
Where a landlord’s TSM data suggests they may be at a higher risk of failing to deliver the outcomes of our standards, we will engage with the landlord. We will decide on the most appropriate course of action on a case-by-case basis, but it may be for example that we decide to carry out responsive engagement and/ or carry out an inspection.
We have published an early indication of findings from the TSM Review in our analysis. Where the TSM Review has identified areas where we do not have sufficient data quality assurance we have reflected this in the data published in this document, either by removing the landlord’s results from the publication (see below) or adding information on this in the published data.
TSM collection
All providers registered with us and who own 1,000 or more units of social housing, are expected to complete the TSM return. These returns provide us with the data required for the risk-based, data-driven approach to regulation.
History
The Tenant Satisfaction Measures (TSMs) were developed following the 2020 Social Housing White Paper. Consultation on the measures took place between December 2021 and March 2022. Following this the measures were finalised in September 2022 with the finalisation of the standard, TSM Technical Requirements and TSM Tenant Survey Requirements. These requirements allow for a robust and broadly comparable set of information, with relevant information for comparison such as survey collection method included.
A provisional set of TSM submission guidance notes were shared with the sector in advance of the finalisation of the return. Feedback from landlords was used to finalise the return and guidance notes.
The first collection of the TSM return occurred in 2024 and collected information on RP’s TSMs. This included the approach to collecting the tenant perception measures, the tenant perception measure results and management information measures, including those on building safety, complaints, repairs and decency. Given this is the first year of data analysis, considerations that would often be part of a technical report are included through the headline publication.
Systems
The return is collected via a web-based system called NROSH+. We control the requirements for data input processes, storage, verification, sign-off and extraction of submitted data and produce the statistical releases. Data are either imported or entered directly into the NROSH+ system by RPs.
Communication with data suppliers
We work closely with RPs to ensure there is a common understanding of the data collection requirements throughout the data collection process. Guidance materials are also promoted to users and published on NROSH+.
Quality assurance processes
We do not have oversight of the systems and data quality assurance processes employed by RPs before submitting data to the TSMs. However, we do provide clear guidance and documentation on NROSH+ and subject incoming returns to a series of checks to identify potential quality issues before each data return is signed off.
The final data files that support the statistical releases are only created once all outstanding queries which could materially impact the quality of the published data are resolved. Any returns not meeting our quality standards are excluded from the final datasets.
Details on landlords who have not submitted data or whose data did not meet our quality standards will be published alongside the underlying data for the years this is required.
In 2023-2024 the following providers’ data has been excluded:
-
Guildford Borough Council (43UD) did not submit TSM data for 2023-24. This failure has been included in our consumer grading.
-
Castle Point Borough Council (22UE) confirmed it had not collected tenant perception information in 2023/24 and is therefore not included. This failure has been included in our consumer grading. Management information has also been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication.
-
Octavia Housing’s (L0717) management information has been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. We will use this information in our ongoing engagement with the landlord.
-
West Northamptonshire District Council’s (5130) management information has been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. We will use this information in our ongoing engagement with the landlord.
-
North Yorkshire Council (5179) management information metrics RP01, RP02 and BS03 have been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. Failures relating to knowledge of DHS, repairs and building safety are included in our consumer grading.
-
Durham Aged Mineworkers Housing Association’s (5215) perception survey wording differed materially from the TSM requirements, reducing its comparability to other landlords. We will use this information in our ongoing engagement with the landlord.
-
London Borough of Lewisham (00AZ) management information metrics RP01 and BS05 have been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. We will use this information in our ongoing engagement with the landlords.
-
South Derbyshire District Council (17UK) management information metrics RP01, RP02, BS02, BS03, BS04 and BS05 have been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. We will use this information in our ongoing engagement with the landlord.
-
East Suffolk Council (5070), Canterbury City Council (29UC), Northumberland Council (00EM) and Arun District Council (45UC) management information metric RP01 has been excluded from the analysis as sufficient data quality assurance could not be obtained in time for publication. We will use this information in our ongoing engagement with the landlords.
For a small number of landlords we have retained data but added a note to aid comprehension of the results. This includes landlords who have confirmed to us that their RP01 denominator is based on the number of units in a stock condition survey, rather than all applicable units for DHS. We are using this information as a source of intelligence for these landlords, including in our ongoing engagement where applicable.
Submission checks
TSM data submitted to us is subject to both automated validation checks and manual inspection.
Automated validations are programmed into the NROSH+ system and check the data at the point of submission for correct formatting, consistency and logical possibility (within expected limits). For example, ensuring numbers of units are consistent across different parts of the return. Automated validations are either ‘hard’ or ‘soft’.
Hard validations – RPs cannot submit without the issue being resolved (for example, if the number of surveys completed is higher than the number of tenants reported by the landlord).
Soft validations – RPs can submit but are required to check their information, and if correct to submit a supporting document (for example, when a value appears to be outside of a normally expected range, such as higher than expected maximum timescale for repairs).
Manual inspections are systematically undertaken on all data submitted. Returns are checked for basic consistency and likely errors, for example where information on emergency appears likely to be based on days rather than hours.
Where we identify a potential anomaly with the submitted data, a query is raised with the submitting RP.
Submission checks and sign-off
It should be noted that the process of signing off data is distinct from our ongoing regulatory work, including the TSM Review outlined above. The sign-off of data confirms that we have investigated areas of potential data error, as highlighted in the validation and checking work, and accepted that the provider has submitted data they believe to be accurate. However, during the course of our regulatory activity, the data will be reviewed alongside other evidence and we may subsequently challenge TSM data with providers. As such, the sign-off and publishing of these data does not constitute our agreement that the provider has met our standards.
Misreporting
There are no known numerical measures of misreporting of data by RPs. However, one source of possible incomparability is inconsistent interpretation of guidance with providers not applying this consistently across the sector. This issue is most likely to arise where there are technical or legal definitions that are complex or, to some degree, ambiguous.
Corrections
Where errors in the submitted data are discovered within a survey year, either through regulatory activity or through provider contact, we allow providers to resubmit data through the NROSH+ portal. Returns can be amended until mid-March the year following their launch.
Statistical release methodology
The data presented in the TSM statistics are drawn from the TSM data.
Accounting for missing data
Most questions in the TSM return are mandatory, and all RPs required to make a TSM return submission are expected to submit the return.
Quality assurance of the published statistics
The data, report and other materials are quality assured by analysts within our statistics production team. This process ensures the figures are consistent across the release, and match the raw data submitted through the TSM, each check being signed off and recorded by the responsible statistician when it has been completed.
Revisions
Under the revisions policy errors identified in the underlying TSM data will be investigated, and revised data gathered. Some corrections may be only minor changes to the data, with little or no impact on the published statistics. These changes will be published at the next scheduled release with no specific announcement.
However, if we become aware of substantial errors in the submitted data, statistical process or other methodology and where a major revision to the published data is required a non-scheduled revision of the statistical release will be published. This will include full details of the revisions, clearly marked data amendments and summary tables showing the overall impact of the changes.
We seek to ensure transparency in processes to maximise user confidence in the quality of our statistical releases.
Why not have your say on our statistics in 2024/25?
We want to hear your views on how the format and range of documents in this statistical release meet your needs. Please complete the form or email feedback, including suggestions for improvements to enquiries@rsh.gov.uk or click below to quickly rate how this document meets your needs.
Our statistical practice is regulated by the Office for Statistics Regulation (OSR).
OSR sets the standards of trustworthiness, quality and value in the Code of Practice for Statistics that all producers of official statistics should adhere to.
You are welcome to contact us directly with any comments about how we meet these standards.
Alternatively, you can contact OSR by emailing regulation@statistics.gov.uk or via the OSR website,
1: Results for those landlords who crossed the 1,000 unit of social housing threshold during the reporting year have not been included in these results or analysis. ↩
2: Low cost rental accommodation and low cost home ownership accommodation are both defined in the Housing and Regeneration Act (2008).↩