Official Statistics

Appendix B - Limitations - Community Life COVID-19 Re-contact Survey 2020

Published 8 December 2020

Applies to England

A number of limitations should be considered when interpreting findings from this report.

We are unable to infer causation. Although this report details the level and dynamics of change between the pre COVID-19 pandemic period and the period during the pandemic, we cannot infer that the pandemic itself caused the change. We lack a reliable estimate of the counterfactual (i.e. what would have happened had there been no pandemic), which is usually necessary to ascribe causality to a particular event or intervention. That said, there are a number of departures from what we might expect that are most plausibly ascribed to the pandemic and its repercussions.

Mode variation between waves. It is important to note that respondents were invited to complete wave 2 online or via telephone, while at wave 1 respondents were invited to complete online or via paper questionnaire. Telephone interviewing was used over paper questionnaires as it was important to complete fieldwork for wave 2 in as narrow a timeframe as possible (given the dynamics of the pandemic). Sending out paper questionnaires and waiting for their return would have extended fieldwork considerably relative to telephone interviewing. Consequently, wave 1 paper respondents completed wave 1 and wave 2 in different modes (of those who completed by paper at wave 1, 77% (n=1,140) completed online and 23% (n=337) by telephone at wave 2).^{[footnote 1]} This may affect some gross change analyses if measurement effects vary between modes.

Question wording variation between waves. There were some minor changes in question wording between wave 1 and wave 2 which affect the ability to make direct comparisons between waves. Some questions were also adapted to capture respondents’ behaviours in relation to COVID-19 and are therefore not strictly comparable.

Question coverage varied by mode. For pragmatic reasons, some questions were not asked across all modes. Some questions were too complicated to be asked via paper questionnaire at wave 1 and some questions were too lengthy to be administered by telephone interview at wave 2.

The following measures were only included online in wave 1:

• Number of hours spent formal volunteering

• Number of hours spent informal volunteering

• Charitable causes donated to.

The following measures were only included in quarter two of the 2018-19 wave 1 survey and were also only included online:^{[footnote 2] [footnote 3]}

• How often do you feel that you lack companionship?

• How often do you feel left out?

• How often do you feel isolated from others?

The follow measures were only included online at wave 2:

• Donation to charity in the past four weeks

• Methods of giving to charitable causes

• Amount given to charity

• Charitable causes donated to

• Factors that would encourage respondents to give more.

Therefore, these measures have smaller base sizes compared to the overall wave 2 sample size. Any exclusions are made clear in the base description in the tables and charts. Weights have been calculated to account for these exclusions. All reported data are weighted to ensure that they are representative of the English population.

Recall period varied by wave. The recall period for the volunteering measures was amended from the original 12-month recall period (at wave 1) to 4 months (at wave 2).^{[footnote 4]} This means wave 2 respondents were asked to recall their volunteering activities and behaviours over a shorter period than the in wave 1. Consequently, some volunteering measures covered in this report are not directly comparable with their wave 1 counterparts. As the wave 2 recall period (4 months) is shorter than the wave 1 recall period (12 months), the wave 2 survey will not identify those who volunteered very infrequently, that is less than once every 4 months but at least once in the last 12 months. This means that overall rates of volunteering cannot be compared on a like-for-like basis for wave 1 and wave 2. Note that this limitation only affects the gross change measure which is based on overall rates of volunteering. Monthly rates of volunteering are unaffected by the change in recall period.

Unobserved differences. Although the weighting protocol makes the wave 2 sample approximately equivalent in observable respects to the wave 1 sample, there may be unobserved (and therefore uncorrected) differences between the two that affect comparability. This risk exists across all surveys. The risk here is small but not negligible.

Comparisons with CLS. Finally, one useful check on the wave 2 data would be to compare the weighted cross-sectional data with comparable data collected from the standard CLS from the same period (such as the first batch of the 2020-21 Quarter 2 sample release). Significant differences between the two may be due to measurement effects or due to uncorrected selection effects (it is impossible to properly distinguish the two) but would at least pinpoint variables where estimates of longitudinal change should be treated with caution. This analysis has not yet been carried out because the 2020-21 CLS data is not yet available.

1. Almost every wave 2 respondent who had completed wave 1 online also completed wave 2 online (2,329 vs. 2,335). ↩

2. ‘No additional weight has been computed for the measure of loneliness. The reason for this is twofold: (i) the wave 2 sample size for any quarter in the baseline wave 1 dataset is too small to reliably evaluate response probabilities separate from the other quarters, and (ii) the quarter 2 2018-19 wave 1 dataset is itself more or less a random subset of the baseline wave 1 dataset so there is no reason to think the wave 2 respondents from this quarter are particularly different from the rest.’ ↩

3. These questions were included in the 2019-20 wave 1 survey on both the online and paper questionnaires. ↩

4. The possible recall period for wave 2 covered the 7th March to the 26th July. ↩