Consultation outcome

Annex 3 - Draft TSM survey requirements (accessible version)

Updated 21 September 2022

Applies to England

1. Introduction

1 - The Tenant Satisfaction Measures Standard requires all registered providers ^{[footnote 1]} to generate and report Tenant Satisfaction Measures as specified by the regulator. This document, Tenant Satisfaction Measures: Tenant Survey Requirements, sets out the basis upon which providers are required to conduct tenant perception surveys to generate a subset of TSMs (‘tenant perception measures’). This document must be read in conjunction with Tenant Satisfaction Measures: Technical Requirements.

2 - The requirements in this document aim to establish a robust basis for generating comparable tenant perception measures across providers. They represent a consistent approach to key aspects of survey methodology, while allowing sufficient flexibility for providers to meet requirements in a way that fits with wider objectives and a range of operating models. This document includes some statistical requirements which are necessarily technical, although it describes how the approach to meeting these requirements should be proportionate to the size and complexity of each provider.

3 - This document is not intended to offer guidance on how to conduct tenant perception surveys. Providers must determine the most appropriate way to meet the requirements in this document, taking advice where necessary.

4 - This remainder of this document is structured as follows:

• Section 1 – Scope of tenant perception surveys
• Section 2 – Sampling approach.

5 - The Annexes contain reference information relevant to our requirements. This includes a summary of requirements for providers with fewer than 1,000 dwelling units of relevant social housing stock (small providers) ^{[footnote 2]} (Annex A), a description of common sampling methods (Annex B), and illustrative sample sizes (Annex C).

6 - These requirements have been prepared by the regulator drawing on advice commissioned from BMG Research.

2. Scope of tenant perception surveys

7 - This section sets out the scope of tenant perception surveys that registered providers are required to undertake with tenants in order to generate tenant perception measures.

Type of survey

8 - The tenant perception measures listed below (TP01-TP12) must be generated using data from perception surveys only. A perception survey is intended to periodically capture tenants’ general views of landlord performance and is not triggered by a recent interaction with the landlord. Providers must not use any data from transactional surveys to calculate these TSMs ^{[footnote 3]}.

Table 1: Type of survey

	Tenant Satisfaction Measure	Collection method
TP01	Overall satisfaction	Perception survey
TP02	Satisfaction with repairs
TP03	Satisfaction with time taken to complete most recent repair
TP04	Satisfaction that the home is well maintained and safe to live in
TP05	Satisfaction that the landlord listens to tenant views and acts upon them
TP06	Satisfaction that the landlord keeps tenants informed about things that matter to them
TP07	Agreement that the landlord treats tenants fairly and with respect
TP08	Satisfaction that the landlord keeps communal areas clean, safe and well maintained
TP09	Lead option: Satisfaction that the landlord makes a positive contribution to neighbourhoods Alternative option: Satisfaction with neighbourhood as a place to live
TP10	Satisfaction with the landlord’s approach to handling of anti-social behaviour
TP11	Satisfaction with the landlord’s approach to handling of complaints
TP12	Tenant knowledge of how to make a complaint

9 - The remainder of this document sets out how providers must undertake perception surveys to generate the tenant perception measures.

Survey questions and response options

10 - Perception surveys must include the survey question wording and response options as specified in Table 2 below. This is irrespective of the survey collection method used (see section 2), so for example in a face-to-face or telephone survey, the surveyor must read the question followed by the response options specified below in full. Written or online surveys must likewise include the response options below. Unless they are necessary to overcome specific barriers to a particular group of tenants completing the survey, providers must not introduce additional visual features such as emojis alongside the response options.

11 - Providers must follow the response scale for each question below. For example, a ‘not applicable/ don’t know’ response must be included where this is shown for a question (and must not be included where this is not shown). For all questions if tenants offer an unprompted ‘not applicable/don’t know’ response, for example during a telephone interview, providers must record this as an effective non-response to this particular question.

Table 2: Tenant perception survey question wording

TSM		Survey question wording
TP01	Overall satisfaction	‘Taking everything into account, how satisfied or dissatisfied are you with the service provided by [your landlord]?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP02	Satisfaction with repairs	‘Has [your landlord] carried out a repair to your home in the last 12 months?” - Yes - No If yes, “How satisfied or dissatisfied are you with the repairs service you have received to your home from [your landlord] over the last 12 months?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied Fairly dissatisfied - Very dissatisfied.
TP03	Satisfaction with time taken to complete most recent repair	‘Has [your landlord] carried out a repair to your home in the last 12 months?” - Yes - No If yes, ‘How satisfied or dissatisfied are you with the time taken to complete your most recent repair after you reported it?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP04	Satisfaction that the home is well maintained and safe to live in	‘Thinking specifically about the building you live in … How satisfied or dissatisfied are you that [your landlord] provides a home that is well maintained and safe for you to live in?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP05	Satisfaction that the landlord listens to tenant views and acts upon them	‘How satisfied or dissatisfied are you with the extent to which [your landlord] listens to your views and acts upon them?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP06	Satisfaction that the landlord keeps tenants informed about things that matter to them	‘How satisfied or dissatisfied are you with the way [your landlord] keeps you informed about things that matter to you as a tenant?’ Response options: - Very satisfied; - Fairly satisfied; - Neither satisfied nor dissatisfied; - Fairly dissatisfied; - Very dissatisfied; - Not applicable/ don’t know.
TP07	Agreement that the landlord treats tenants fairly and with respect	‘To what extent do you agree or disagree with the following “my landlord treats me fairly and with respect”?’ Response options: - Strongly agree - Agree - Neither agree nor disagree - Disagree - Strongly disagree - Not applicable/ don’t know.
TP08	Satisfaction that the landlord keeps communal areas clean, safe and well maintained	‘Do you live in a building with communal areas, either inside or outside, that you share with other people who live in the building? - Yes - No If yes, ‘How satisfied or dissatisfied are you that [your landlord] keeps these communal areas clean, safe and well maintained?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP09	Satisfaction that the landlord makes a positive contribution to neighbourhoods	Lead option: Thinking about what your landlord does to improve your neighbourhood as a place to live…‘How satisfied or dissatisfied are you with the extent to which [your landlord] makes a positive contribution to your neighbourhood?’ Alternative option: ‘How satisfied or dissatisfied are you with your neighbourhood as a place to live?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP10	Satisfaction with the landlord’s approach to handling of anti-social behaviour	‘How satisfied or dissatisfied are you with [your landlord]’s approach to handling anti-social behaviour?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP11	Satisfaction with the landlord’s approach to handling of complaints	How satisfied or dissatisfied are you with [your landlord]’s approach to complaints handling?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - - Not applicable/ don’t know.
TP12	Tenant knowledge of how to make a complaint	‘To what extent do you agree or disagree with the following ‘I know how to make a complaint to [my landlord] if I am not happy with the service I receive’?’ Response options: - Strongly agree - Agree - Neither agree nor disagree - Disagree - Strongly disagree - Not applicable/ don’t know.

Questionnaire structure

12 - Registered providers must ensure the purpose of the perception survey is explicitly communicated to the tenant at the start of each survey conducted. Specifically, alongside any reference to other purposes, providers must state that the survey will be used to calculate annual TSMs to be published by the landlord.

13 - In addition to the questions above, providers are permitted to include other questions in the same tenant perception survey questionnaire. These additional questions may be used, for example, to better understand perceptions on wider or more detailed aspects of performance.

14 - The overall satisfaction question to generate the overall satisfaction TSM (TP01) must appear as the first question in any perception survey questionnaire used to generate TSMs. Questions to generate the other TSMs must precede any other question(s) that a provider chooses to include on the same topic. This means that the questions to generate these TSMs (TP01 - TP12) may appear together at the start of a survey or separately at the start of relevant sections with related questions. For example, if the survey includes a section with several questions on repairs, the TSM questions on repairs (TP02 - TP03) must appear before any other questions on repairs – either at the start of this specific section or at the start of the survey. Beyond these requirements, there is no prescribed order for the questions.

15 - Providers must take reasonable action and care when undertaking or commissioning surveys to ensure that participants are not led toward a particular point of view. In particular, beyond any necessary clarification, providers must avoid any additional wording or questions preceding a question that is likely to have a significant impact on responses.

16 - Providers must consider the most appropriate length of the survey in order to maximise response rates across the range of households, given the information the provider needs to generate, and to avoid survey fatigue having a significant impact on responses. Providers must advise respondents of the approximate length of completion at the start of the survey.

Relevant tenant population

17 - Providers must use the relevant tenant population as set out in Table 3 as the basis for tenant perception measures. This means that all households in the relevant tenant population must be included within the sampling frame to generate a perception measure [^4 ]. For providers that own at least 1,000 relevant dwelling units of relevant social housing stock, this means that:

• Providers that own 1,000 or more Low Cost Rental Accommodation dwelling units must report tenant perception measures for LCRA specifically. For these perception measures, the relevant population comprises all households residing in LCRA owned by the provider.

• Providers that own 1,000 or more Low Cost Home Ownership units must report tenant perception measures for LCHO specifically. For these perception measures, the relevant population comprises all households residing in LCHO owned by the provider.

• Providers that own 1,000 or more LCRA dwelling units, but fewer than 1,000 LCHO dwelling units, are only required to report tenant perception measures for LCRA (and vice versa).

• Providers that own 1,000 or more LCRA dwelling units and 1,000 or more LCHO dwelling units must report tenant perception measures for LCRA and LCHO separately.

18 - Registered providers with fewer than 1,000 dwelling units of LCRA and fewer than 1,000 dwelling units of LCHO can choose to calculate and report perception measures for LCRA (only), LCHO (only), both stock types separately, or both stock types combined. These providers must be able to demonstrate a proportionate and rational approach to determining the basis for reporting perception measures in this respect, following their stock profile. The relevant tenant population must follow the basis such a provider selects to report perception measures. For example, if a provider reports perception measures for LCRA and LCHO combined, the relevant tenant population must comprise all households residing in LCRA and LCHO owned by the provider.

Table 3: Relevant tenant population

Tenant perception measures	Relevant tenant population	Providers required to report tenant perception measures on this basis
Tenant perception measures for LCRA	All households residing in LCRA owned by the provider.	All providers that own 1,000 or more LCRA dwelling units.
Tenant perception measures for LCHO	All households residing in LCHO owned by the provider.	All providers that own 1,000 or more LCHO dwelling units.
Tenant perception measures for LCRA and LCHO combined	All households residing in LCRA and LCHO owned by the provider.	Providers that own fewer than 1,000 dwelling units of LCRA and fewer than 1,000 dwelling units of LCHO are can choose to report perception measures for LCRA (only), LCHO (only), LCRA and LCHO separately, or LCRA and LCHO combined.

Units of measurement

19 - In order to generate tenant perception measures and assess statistical accuracy, providers must record the number of responses to each survey question and the total responses to the perception survey as a whole. For each question, there must be a maximum of one response recorded per household irrespective of how many legal tenants there in a household. For the purposes of these requirements, a household refers to the group of people that occupies a single dwelling unit (as it is defined in Tenant Satisfaction Measures: Technical Requirements ^{[footnote 5]} ).

20 - Providers must record the number of responses to each question as the total number of surveyed households that have provided one of the following responses: Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied ^{[footnote 6]}. Providers must ensure that the count of responses for a question does not include any households that have either answered ‘Not applicable/don’t know’ or otherwise have not responded to the question.

21 - Providers must include partial survey responses within the count of responses – for example, if a household has responded to some but not all questions in the survey, they must be included within the responses for those questions they have responded to.

22 - Providers must record the total responses to the tenant perception survey as the total number of surveyed households that have provided one of the following responses – Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied (or the equivalent agreement responses) – to at least one of the tenant perception survey questions.

Survey timings

23 - Data used to generate tenant perception TSMs must be collected as part of a single integrated survey exercise which has been designed to meet the requirements in this document. This requires that all survey responses used by a provider must be generated using a questionnaire consistent in length and structure, an integrated sampling approach (e.g. to avoid double counting of households) and be subject to an overarching assessment of statistical accuracy and representativeness which meets the requirements set out in this document. The fieldwork undertaken as part of this exercise may happen at a single point in time, through a phased approach, or on a periodic rolling basis (e.g. quarterly or monthly). Informal survey data which does not meet the requirements set out in this document must be excluded from the responses used to calculate the perception measures.

Providers that own 1,000 or more dwelling units of relevant social housing stock ^{[footnote 7]} must carry out tenant perception surveys of the relevant tenant population (as specified in Table 3) at least annually. This means that tenant perception measures for these providers must only be calculated using responses from surveys conducted within the reporting year the TSMs relate to. The reporting year for all providers with 1,000 or more dwelling units of relevant social housing stock is 1 April to 31 March.

24 - Providers that own fewer than 1,000 dwelling units of relevant social housing stock (small providers) must carry out perception surveys at least once every two years ^{[footnote 8]}. This means that tenant perception measures for these providers must only be calculated using responses from surveys conducted within the past two reporting years. Providers with fewer than 1,000 dwelling units of relevant social housing stock are permitted to collect and report annual TSMs according to a reporting year other than 1 April to 31 March. For example, small providers are permitted to use a reporting year for TSMs which aligns with their particular financial reporting year.

25 - In calculating the perception measures, providers must use the most recent survey exercise completed that meets the requirements in this document.

In-house and outsourced surveys

26 - Providers must ensure that they employ a suitable level of expertise to design and apply a survey methodology to generate tenant perception measures in a way that meets the requirements of this document. The level of expertise required will be proportionate to the size and complexity of each provider’s tenant base and sampling approach chosen. For example, the level of required statistical and market research expertise is greater for a large provider required to ensure representativeness of responses across a diverse tenant base, than for a provider with fewer than 1,000 dwelling units undertaking a relatively simple census across all relevant tenant households.

27 - In meeting the requirements of this document, providers can choose to conduct surveys and calculate the perception measures in-house, or through external contractors. Providers must ensure that any external contractor employed in this capacity adheres to the Market Research Society Code of Conduct where its provisions are relevant to tenant perception surveys described in this document. It is ultimately the responsibility of Boards of private registered providers and governing bodies of local authority registered providers to ensure that the tenant perception TSMs are accurate and built on a survey methodology that meets the requirements set out in this document.

Data protection and confidentiality

28 - Registered providers must ensure that they and any contractors are compliant with the relevant privacy and data protection legislation when processing personal data.

29 - Providers must protect respondent confidentiality in order to protect tenants and give respondents confidence to give honest feedback. This includes ensuring that identifiable individual responses are not shared internally or externally beyond those who require the information to produce the TSMs, unless the respondents give explicit consent, and this is managed appropriately. This applies to both the conduct of the survey itself, and the publication of results.

30 - Providers with fewer than 1,000 dwelling units of relevant social housing stock must take a proportionate approach to tenant perception measures in order to protect tenant confidentiality. For the smallest of these providers, it may be difficult to protect the anonymity of individual tenants if tenant perception information is published in full. It is permissible for at least some tenant perception measures to be omitted from published information if providers judge this to be a material risk.

Publication of summary of approach

31 - Providers must publish a summary of the survey approach used to generate published tenant perception measures. This must be available alongside each set of tenant perception measures published by the provider. It must include a summary of achieved sample size (number of responses); timing of survey; collection method(s); sample method; assessment of representativeness of the sample against the relevant tenant population; any weighting applied to generate the reported perception measures; and any other methodological issues likely to have a material impact on the tenant perception measures reported.

32 - This summary must state if the provider has undertaken any tenant perception surveys which include TSM questions but has not included these responses in the calculation of the TSMs. A rationale for why this information has been excluded must be provided. The provider must include a summary of responses by survey collection method and the rationale for the survey collection method(s) chosen. As part of this summary, providers must publish the questionnaire used to generate survey responses including any explanatory wording communicated to tenants.

33 - This summary must be proportionate to the complexity of the sampling methods employed and must include sufficient information to enable reasonable assessment of the validity of the published tenant perception measures. For example, the level of detail required from a relatively large provider applying stratified sampling and weighting of responses is significantly greater than that required from a small provider employing a simple census approach.

Regulatory data returns

34 - Registered providers must submit to the regulator information relating to TSMs as specified by the regulator. We expect that this will, in general, include a standard data return prepared by the regulator covering both calculated tenant perception measures as well as summary information on the survey approach used.

3. Sampling approach

35 - This section sets out the approach providers must take to generating a sample of responses used to calculate tenant perception measures. There are two broad approaches to collecting such a sample:

• A census approach – where all households in the relevant tenant population are invited to participate in the survey.

• A sample approach – where a sample of relevant tenant households are invited to participate in the survey.

36 - With either approach, the accuracy of tenant perception measures depends on how many households respond to the survey (‘the sample size’) and how well these respondents represent the relevant tenant population (‘representativeness’).

Sample size

37 - When undertaking a survey, providers must, as far as possible, generate a sample size for overall satisfaction (TP01) that meets the minimum level of statistical accuracy set out in Table 4 ^{[footnote 9]}. For example, a provider with 10,000 LCRA dwelling units must, as far as possible, achieve a sample where estimated overall satisfaction for LCRA tenants is accurate to a margin of no more than +/-3% at a 95% confidence level. With this level of accuracy, the provider can be 95% sure that – as long as the sample is representative – the estimated level of satisfaction is right (to within 3% above or below) for the whole population. ^{[footnote 10]}

Table 4: Required minimum levels of statistical accuracy for overall satisfaction

Population	Required minimum statistical accuracy (margin of error at 95% confidence level)
Fewer than 2,500 dwelling units	+/- 5% [footnote 11]
2,500 – 9,999 dwelling units	+/- 4%
10,000 – 24,999 dwelling units	+/- 3%
25,000 dwelling units or more	+/- 2%

38 - Providers must determine the sample size required to achieve the required levels of statistical accuracy, for example using an online calculator or expert statistical input ^{[footnote 12]}, in order to plan survey activity. The sample size required to achieve a given level of statistical accuracy applies to each relevant tenant population for which separate perception measures are required ^{[footnote 13]} . The required sample size depends primarily on the size of this relevant tenant population. Annex C sets out indicative sample size corresponding to example relevant tenant populations.

39 - The sample size refers to the total number of responses to the tenant perception survey. ^{[footnote 14]} It is not the number of surveys issued or contacts attempted but the number of responses received. Since precise response rates are unknown in advance, providers must incorporate an appropriate degree of contingency in their planned survey activity in order that the achieved sample is likely to be at least the size required to meet the minimum level of statistical accuracy set out in Table 4. ^{[footnote 15]}

40 - Providers that own 1,000 or more dwelling units of relevant social housing stock ^{[footnote 16]} must as far as is possible meet these minimum standards of statistical accuracy – whether they undertake a census or other sampling approach – unless there are exceptional circumstances that mean this is not practically possible using standard survey approaches. Providers may choose to exceed minimum sample sizes, for example to generate robust estimates of satisfaction for particular groups of tenants.

41 - Where it is practically difficult to achieve minimum sample sizes using conventional survey collection methods, providers may undertake a census as a means of meeting the requirements of this document in relation to sample size (i.e. inviting all households in the relevant tenant population to participate in a survey). This is irrespective of whether a census approach generates the minimum level of statistical accuracy set out in Table 4. This is likely to apply in particular to small providers with fewer than 1,000 dwelling units.

Representativeness of responses

42 - A survey that has met the sample size requirements can still be biased if groups of tenants who on average have higher satisfaction are over-represented in the sample (or vice versa).

43 - Providers must ensure that, as far as possible, survey responses used to calculate perception TSMs are representative of the relevant tenant population. ^{[footnote 17]} Providers can meet this requirement through one of two routes:

a) A representative sample: This means there is no material under – or over – representation of tenant groups (compared to the relevant tenant population) that is likely to affect calculated satisfaction scores. Using this approach, providers must ensure that the achieved sample is representative of the relevant tenant population.

b) Weighting responses: If the achieved sample is not representative of the tenant population then providers must appropriately weight the responses to ensure the TSMs reported are representative. ^{[footnote 18]}

44 - All providers must assess the extent to which the achieved sample is representative of the relevant tenant population. Specifically, providers must undertake reasonable checks for differences between total survey responses and the relevant tenant population in terms of characteristics associated with different average satisfaction scores. For example, for many providers tenants in sheltered housing or in larger properties typically have higher satisfaction than average – for large providers where this is likely to have a material impact on satisfaction scores, reasonable checks are likely to include assessing how the proportion of survey responses from such households compares to the relevant tenant population (see illustration in Table 5 below).

Table 5: Table 5: Illustration of a check for representativeness of survey responses

	Relevant tenant population	Total survey responses	Calculated satisfaction score
	(dwelling units,% total)	(% total)
Provider 1
LCRA	25,000 (100%)	2,500 (100%)	75%
Check: Housing type
Sheltered housing	7,500 (30%)	1,250 (50%) X	85%
General needs housing	17,500 (70%)	1,250 (50%) X	65%
Provider 2
LCRA	50,000 (100%)	5,000 (100%)	71%
Check: Housing type
Sheltered housing	15,000 (30%)	1,500 (30%) V	85%
General needs housing	35,000 (70%)	3,500 (70%) V	65%

X - In this example provider 1’s sheltered housing (30% of all units) is over-represented in the sample (50% of responses) and general needs housing (70% of units) is under-represented (50% of responses). As there is a large difference in the satisfaction score recorded (85% in sheltered housing, 65% in general needs housing) it is likely to have a material impact on the calculated satisfaction score. Therefore, the sample is not representative and the perception measures will be biased estimates of the satisfaction score for the relevant tenant population. To address this, responses must be appropriately weighted to ensure the calculated perception measures are representative of the relevant tenant population

V - Provider 2 has surveyed a representative proportion of tenants based on stock type. 30% of tenants live in sheltered housing and 30% of those who responded live in this stock type. Similarly, 70% of tenants live in general needs housing and 70% of those who responded live in this stock type. Provider 2 does not need to weight its responses based on stock type.

This illustration focusses on stock type. Providers need to consider what other characteristics should be included in their assessment of representativeness.

45 - Providers must reach a balanced judgement as to which characteristics to include in this assessment of representativeness based on their particular tenant profile, evidence or rationale for potential different satisfaction scores by characteristic, and available data. Current evidence from providers’ analysis of tenant satisfaction suggests that the following characteristics are material for many large providers:

• stock type (e.g. general needs, housing for older people/sheltered housing, other supported housing, temporary social housing)

• age of respondent

• ethnicity

• building type (e.g. high rise/flats)

• property or household size

*geographical area or estate.

46 - For most large providers, stock type is likely to be a significant influence on satisfaction scores and is likely to be included within the scope of an assessment that meets this requirement.

47 - The scope of the assessment must be proportionate to the size and complexity of each provider’s stock, tenant base and sampling approach. Providers with fewer than 1,000 dwelling units of relevant social housing stock (small providers) ^{[footnote 19]} are only required to undertake a high-level assessment of representativeness. For example, by checking their average results against a small number of key characteristics.

48 - If this assessment confirms the sample is not representative, then providers must appropriately weight the responses to ensure the TSMs reported are representative. Weighting needs to be conducted appropriately to meet this objective and may require expert input. Weighting used to calculate perception TSMs must not be undertaken to meet other objectives, for example to attempt to control for potential effects of different collection methodology or to seek to reflect a national tenant profile. Weights must be calculated on the basis of all responses to the perception survey and applied consistently across all tenant perception measures.

49 - Providers with fewer than 1,000 dwelling units of relevant social housing stock are not required to weight responses unless there is strong evidence of a significant bias in estimated scores.

50 - Providers are permitted to sample a larger proportion of tenants in any particular group, either due to a known issue of low response rates or to gain more detailed statistically valid information. Providers that choose to ‘over-sample’ particular groups must assess the impact on the representativeness of the sample and must weight results to ensure representativeness when generating TSMs for the overall relevant tenant population.

Survey collection method

51 - Providers must use an appropriate survey collection method considering factors such as likely response rate, cost, addressing barriers to participation, tenant profile, and the representativeness of responses. As long the requirements of this document (and other applicable requirements) are met, providers can use any standard collection method including face-to-face, telephone, postal, or email/online delivery and may use more than one collection method.

52 - Providers must be able to demonstrate a rationale for the survey collection methods chosen with reference to the size and characteristics of their relevant tenant population. In particular, survey collection methods where the staff or contractors ask tenants the questions and report their response (e.g. face-to-face, telephone) can generate higher satisfaction scores than other methods. They may however be important to address barriers to participation, meet minimum sample size requirements, enable richer information to be generated, and for many small providers may be more cost-effective than setting up postal or online surveys.

Sampling method

53 - Providers must use a sampling method consistent with the requirement of this document, in particular that survey responses used to generate perception measures are representative of the tenant population as a whole. The following are examples of common sampling methods that are generally consistent with this requirement: computer-generated random sampling, systematic sampling, stratified sampling, or a census approach. In contrast, cluster sampling carries significant risks of generating a non-representative sample and must only be used where the provider has a strong practical rationale for adopting this method and can demonstrate that the risk is fully understood and addressed. Common sampling methods are described in Annex B.

Accessibility and barriers to responding

54 - Providers must take reasonable steps to assess, identify and remove barriers to certain groups of tenants participating in surveys. In particular, this is in respect to tenants who share one or more protected characteristics under the Equality Act 2010, and in respect of duties of that Act. Barriers may include, but are not limited to, language barriers, visual impairment, literacy or lack of access to digital media. Where necessary to overcome barriers to participation, it is permissible for surveys to be completed by a carer or another household member on behalf of a tenant.

55 - Under exceptional circumstances, providers are permitted to remove certain households from the sample frame where there are significant capacity issues or health and safety risks that cannot be reasonably surmounted. This may apply to particularly vulnerable tenants in specific types of supported housing for example.

56 - Providers can choose to incentivise tenants to encourage survey completion (e.g. by including all survey participants in a prize draw). When considering the use of incentives, providers must consider the potential impacts on the representativeness of responses.

Annex A – Summary of requirements for small providers

57 - For the purposes of this document, small providers are those which own fewer than 1,000 dwelling units of relevant social housing stock. ^{[footnote 20]}

58 - All requirements set out in this document apply to small providers unless stated otherwise. This section summarises the provisions set out above that specifically apply to small providers. These recognise the practical barriers for small providers in conducting surveys to the same statistical accuracy and detail as larger providers, and the likelihood that many small providers will need to employ a census survey approach to meet the requirements of this document.

Survey frequency

59 - Small providers must carry out perception surveys at least once every two years. This reduced frequency recognises the greater relative costs and capacity constraints for small providers in conducting a census of all households and/or achieving relatively high response rates consistent with minimum levels of statistical accuracy. Small providers are permitted to collect and report annual TSMs according to a reporting year other than 1 April to 31 March.

Relevant tenant population

60 - Small providers must take a proportionate approach to defining the relevant tenant population depending on their stock profile. Small providers can choose to report tenant perception measures for households in LCRA, LCHO stock, or both stock types combined.

Sampling approach

61 - Small providers must, as far as possible, generate an estimate for overall satisfaction accurate to a margin of no more than +/-5% at a 95% confidence level. While this is a wider margin than for many large providers, it may still be practically difficult to achieve for many small providers. In this instance, small providers can choose to undertake a census. A small provider that undertakes a census will be considered to have met the requirement for statistical accuracy, irrespective of the number of responses they achieve.

62 - All providers must assess the extent to which the achieved sample is representative of the relevant tenant population. However, the scope of this assessment must be proportionate to the size and diversity of each provider’s tenant base and sampling approach. Small providers are only required to undertake a high-level assessment of representativeness of the achieved sample against the relevant tenant population, for example against a small number of key characteristics, and are not required to weight responses unless there is strong evidence of a significant bias in estimated scores.

Confidentiality

63 - Small providers must take a proportionate approach to tenant perception measures in order to protect tenant confidentiality. For the smallest of these providers, it may be hard to protect the anonymity of individual tenants if tenant perception information is published in full. It is permissible for at least some tenant perception measures to be omitted from published information if providers judge this to be a material risk.

Annex B – Description of common sampling methods

Computer-generated random sampling

64 - A list of all the relevant tenant population is produced and the required number of households to be surveyed are randomly selected, for example using a computer-generated random sampling.

Systematic sampling

65 - A list of all the relevant tenant population is produced and every nth household is selected for sampling. For example, a provider with 10,000 units decides to sample 2,000 properties. ^{[footnote 21]} For a required sample of 2,000 of 10,000 households, every fifth (10,000 / 2,000) household is selected for surveying.

Stratified sampling

66 - The relevant tenant population is divided into separate groups called strata based on characteristics of interest and then a proportionate sample is drawn from each group using random or systematic sampling. Strata may include, for example, stock types (general needs and supported housing), geography (schemes, local authorities) or tenant characteristics. For a provider with 10,000 units, of which 8,000 (80%) are general needs and 2,000 (20%) are supported housing, using stratified sampling to generate a proportionate sample of 2,000 responses across stock types would mean as far as possible generating a sample of 1,600 responses from general needs households (2,000 x 80%) and 400 in supported housing (2,000 x 20%).

Cluster sampling

67 - The relevant tenant population is divided into separate groups, called clusters, such as by street or postcode areas. A random sample of clusters is selected from the population, capturing a sufficient number of households to satisfy sample size requirements. There may be practical benefits for some survey methods (e.g. face-to-face), but this method relies on chosen clusters being sufficiently representative of the population as a whole.

Census approach

68 - An entire population of relevant tenant households is invited to participate in the perception survey, although typically not all invited to participate will respond. This is commonly used by small providers where it may be challenging to achieve a minimum sample size associated with standard levels of statistical accuracy.

Annex C – Illustrative sample sizes

69 - Providers must determine the sample size required to achieve the required levels of statistical accuracy. The table below provides an illustration of achieved sample sizes required to ensure statistical accuracy based on example population sizes, in order to give providers a sense of scale for what achieved sample sizes will be required. There are numerous free-to-use online calculators that can generate sample sizes from known population sizes using a standard formula consistent with the table below ^{[footnote 22]}.

70 - Indicative sample sizes in this table are consistent with an estimated satisfaction score of 50%. While a higher or lower satisfaction score would reduce the number of responses required, there is a risk that assumptions over average satisfaction before a survey has been conducted are inaccurate, especially as providers transition to the methodology set out in this document to generate satisfaction scores. Using an assumed 50% satisfaction score allows for a degree of contingency in planning survey activity.

71 - The indicative sample sizes in the next table assume no weighting of survey responses. Weighting will reduce the achieved sample size (a lower “effective sample size”) and may therefore require a higher number of responses. Estimation of the effective sample size may require expert input.

Annex table: Indicative sample sizes consistent with meeting minimum levels of statistical accuracy for example population sizes

Population (dwelling units/households)	Margin of error at 95% confidence level	Indicative achieved sample size
100	+/- 5%	80
250	+/- 5%	152
500	+/- 5%	218
750	+/- 5%	255
1,000	+/- 5%	278
1,500	+/- 5%	306
2,000	+/- 5%	323
2,500	+/- 4%	485
3,000	+/- 4%	501
4,000	+/- 4%	522
5,000	+/- 4%	536
7,500	+/- 4%	556
10,000	+/- 3%	965
15,000	+/- 3%	997
20,000	+/- 3%	1,014
25,000	+/- 2%	2,191
50,000	+/- 2%	2,291
75,000	+/- 2%	2,327
100,000	+/- 2%	2,345

1. Every reference of ‘registered providers’ and ‘providers’ in this document relates to registered providers of social housing. ↩

2. For the purposes of this document, a small provider is defined as one that owns fewer than 1,000 units of low cost rental accommodation and low cost home ownership dwelling units (combined). For providers in a group, dwelling units must be measured at a registered group level. ↩

3. Transactional surveys are triggered by a recent interaction (such as, by way of example only, completion of a repair, or response to a complaint). ↩

4. This means that a single household refers to the occupant(s) of each self-contained unit or each bedspace in non-self-contained housing. ↩

5. Or equivalently, for TP07 and TP12, one of the following responses: strongly agree, agree, neither agree nor disagree, disagree, strongly disagree ↩

6. That is, 1,000 or more dwelling units of low cost rental accommodation and low cost home ownership dwelling units (combined). For providers in a group, dwelling units must be measured at a registered group level. ↩

7. Providers with fewer than 1,000 dwelling units of relevant social housing stock must still report tenant perception TSMs in their first published set of TSMs. All providers must undertake the necessary perception surveys to allow them to report tenant perception TSMs for the first year that TSMs are to be reported. ↩

8. As the response rates for different questions will vary (for example based on those who choose not to respond to a particular question) this requirement applies specifically to estimated overall satisfaction. ↩

9. Strictly speaking, this requires a sample to be random and representative. ↩

10. Where it is not possible to achieve this level of statistical accuracy (for example for many providers with fewer than 1,000 dwelling units of relevant social housing stock), employing a census approach is sufficient to meet this requirement. ↩

11. here are numerous free-to-use online calculators that generate sample sizes from known population sizes using a standard equation as described in Annex C. ↩

12. For example, a provider with 10,000 LCRA units and 2,500 LCHO units is likely to require total responses for overall satisfaction from approximately 965 LCRA and 485 LCHO tenant households respectively in order to meet the minimum level of statistical accuracy. ↩

13. Total responses to the tenant perception survey is defined in this document as the total number of surveyed households that have provided one of the following responses – Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied (or the equivalent agreement responses) - to at least one of the tenant perception survey questions. ↩

14. The estimated satisfaction score - and any weighting requirements - are also unknown at the outset. See Annex C for more information. ↩

15. That is, 1,000 or more dwelling units of low cost rental accommodation and low cost home ownership dwelling units (combined). ↩

16. Equivalently, this means that providers must ensure that as far as possible calculated perception measures are unbiased estimates of the satisfaction score for the relevant tenant population. ↩

17. Providers with fewer than 1,000 units are not required to weight responses unless there is strong evidence of a significant bias in estimated scores. ↩

18. That is, providers that own fewer than 1,000 units of low cost rental accommodation and low cost home ownership combined at a registered group level. ↩

19. That is, providers that own fewer than 1,000 units of low cost rental accommodation and low cost home ownership combined. For providers in a group, units are measured at a registered group level. ↩

20. A provider with 10,000 units would need 965 responses to achieve minimum levels of statistical accuracy. Sampling 2,000 properties means they expect a response rate greater than 48%. ↩

21. For the avoidance of doubt, the standard equation for calculating the sample size (n) referred to in these requirements is as follows: n=((z^2×P(1-P))/ε^2 )/ (1+ (z^2×P(1-P))/(ε^2×N)), where z is the Z-score related to the confidence level (1.96 for 95%), P is the estimated satisfaction score, N is the relevant number of units held by the provider and ε is the margin of error for the population size shown in the table. This equation assumes there is no weighting. ↩