Guidance

National Travel Survey: conditional incentive experiment

Updated 16 April 2025

Chapter 1: Background of conditional incentives on the National Travel Survey

Conditional incentives, unlike unconditional ones, are provided only to those who complete specific survey tasks. While unconditional incentives rely on the principle of reciprocity, conditional incentives function as compensation for the respondent’s time and effort, reflecting a transactional exchange. These incentives are expected to increase the perceived benefits of participation, making the cost-benefit analysis more favourable for potential respondents.

On the National Travel Survey (NTS) in England, a £5 voucher has been offered as a conditional incentive to all household members who fully participate in the study by completing both the interview and the seven-day travel diary. This incentive has remained unchanged since conditional incentives were introduced to the survey in the early 2000s. However, due to inflation, the real value of this incentive has substantially declined over time, making it lower than those offered by other large-scale social surveys in the UK (for example, the British Social Attitudes Survey), although in larger households the total value can be higher.

During the second quarter of 2024, an experiment was conducted to explore the efficacy of increasing the value of the conditional incentive. Specifically, the performance of two different conditional incentive values were compared:

• the traditional £5 voucher, offered to each household member if all members fully complete the study

• a higher value £10 voucher, offered under the same conditions

The main indicators used to evaluate the performance of the £10 incentive against the traditional £5 incentive included:

• response rates, which are an important measure of survey performance

• sample composition, which is an indicator of potential response bias

Based on previous research, the following hypotheses were proposed:

H1: Response rates will be higher for households offered the £10 incentive.

This hypothesis was based on a large body of research demonstrating the positive effect of incentives on response rates. The effectiveness of the increased incentive was assessed by comparing the proportion of productive households between the two incentive conditions.

H2: The sample in the £10 condition will more closely resemble the population than the sample in the £5 condition.

This hypothesis was supported by a growing body of literature suggesting that larger incentives can reduce non-response bias by attracting harder-to-reach groups, such as younger respondents and those born outside the country. This hypothesis was tested by comparing the achieved samples for the two incentive conditions with benchmark data from the 2021 Census, using dissimilarity indices.

Chapter 2: Research design

A randomised experiment was conducted during the second quarter (April to June) of 2024. The sample was drawn from the Postcode Address File (PAF) and sorted by postcode. Within each point, half of addresses (11 out of 22) were selected using systematic random sampling. The selected addresses were then assigned to either the traditional £5 incentive condition (the control group) or the £10 incentive condition (the treatment group). Since assignment was made at the address level rather than the point level, interviewers were assigned addresses in both the control group and treatment groups, helping to control for potential interviewer effects.

Chapter 3: Findings

3.1. Response rates

Response rates are commonly used indicators to assess survey performance. Although low response rates do not necessarily lead to response bias, they do increase the risk of it and reduce the available sample size, thereby decreasing statistical power. In this section, the response rates for the two incentive groups are compared with reference to the following two outcome categories:

• fully productive households, where all members completed the seven-day travel diary in addition to the interview

• partially productive households, where some, but not all, members completed the travel diary

These two outcome categories are the most relevant to this analysis because most NTS estimates are based on fully productive households, although partially productive cases are included in some calculations.

To compute the rate of fully productive households, the number of fully responding households was divided by the total number of addresses after excluding deadwood cases. Proportion tests were then computed using the R software package (R Core Team, 2024) to compare whether the rates of fully productive households were comparable between the two incentive groups.

In quarter 2, the proportion of fully productive households was 1.4 percentage points higher in the £10 conditional incentive group, resulting in 33 additional fully completed households over the quarter. However, this difference was small and did not reach statistical significance.

This is shown in Table 1 below, which displays the proportion of fully productive households and then the proportion of partially productive households for each incentive condition, along with the results of statistical testing, namely for the chi-squared test measure (‘Χ²’) and statistical significance (‘p’) used throughout this report. The figures are presented for each of the three months individually and then for the quarter as a whole.

In both April and May, the £10 incentive group outperformed the £5 group by approximately 2 percentage points. In June, however, response rates were almost identical. None of these differences were statistically significant.

Partially productive cases, where not all household members completed the diary, were more common in the £5 condition than in the £10 condition. This difference can be primarily attributed to the results in April, where the proportion of partially productive cases was twice as high in the lower incentive group. Although some limited information from partially productive cases is used, most estimates rely on fully productive cases.

Table 1: Proportion of fully productive and partially productive households by incentive condition

Case Type	Period	£5 incentive	£10 incentive	Χ2	p
Fully productive	April	31.1%	32.9%	0.693	0.405
Fully productive	May	35.1%	37.3%	0.951	0.329
Fully productive	June	34.2%	34.3%	0.000	1.000
Fully productive	Quarter 2	33.4%	34.8%	1.200	0.273
Partially productive	April	8.3%	4.0%	15.242	less than 0.001
Partially productive	May	6.1%	6.0%	0.002	0.961
Partially productive	June	7.1%	6.1%	0.554	0.457
Partially productive	Quarter 2	7.2%	5.4%	7.987	0.005

3.2: Sample composition

An essential goal of any survey is to accurately represent the population from which its sample is drawn. One of the main threats to external validity is non-response bias, which occurs when individuals who participate in the study differ systematically from those who do not. This risk is particularly heightened when response rates are low, as only a small fraction of the invited sample ends up participating.

To identify potential selection biases, a common approach is to compare the characteristics of the sample with benchmark data from the population. In this section, data from fully productive households in the two incentive groups are compared with data from the 2021 Census using dissimilarity indices. For each socio-demographic variable, the dissimilarity index score (referred to as ‘d’) is calculated as the sum of the absolute differences between the sample proportion and the benchmark proportion, divided by half. The dissimilarity index score reflects the proportion of observations that would need to change categories in the experimental groups to perfectly match the benchmark data (Biemer et al., 2018). As a general guide, d with a value of 0.10 or less represents ‘good’ agreement, while a value of 0.05 or less suggests ‘very good’ agreement. The main objective of this report has been to determine which of the two incentive groups better represents the target population before any post-survey adjustments.

Across the three household-level variables, the £10 condition slightly outperformed the £5 condition in tenure (d of 0.11 compared to 0.12) and deprivation (d of 0.09 compared to 0.10), while the £5 condition performed slightly better in household size (d of 0.04 compared to 0.05), as shown in Figure 1 below.

Table 2 further below shows the proportion of households in each category of the tenure, deprivation and household size variables for both incentive conditions, as well as the proportions found in the 2021 Census. Below that, Table 3 presents the results of the statistical testing for the variables analysed.

From here it is clear both incentive groups overrepresented homeowners. Figure 1 shows that these dissimilarity indices are above 0.10, which indicates less than ‘good’ agreement with the benchmark data.

Regarding deprivation, which has been analysed throughout using the 2019 Index of Multiple Deprivation rankings (IMD2019 Rank), both groups achieved ‘good’ agreement (d less than or equal to 0.10). They underrepresented the most deprived areas (deciles 1 to 4) and overrepresented the least deprived areas (deciles 7 to 10). However, the £5 group showed larger deviations, particularly in the most and least deprived areas, resulting in the slightly higher dissimilarity index score (d of 0.10) than the £10 group (d of 0.09).

In terms of household size, both groups performed well (d less than or equal to 0.05), with the £5 group showing a slightly better alignment with the benchmark data (d of 0.04) compared to the £10 group (d of 0.05). Both groups showed only a slight overrepresentation of households with two members.

Figure 1: Dissimilarity index scores for household-level variables, for each incentive condition

Figure 1 shows a bar chart of the dissimilarity index scores for the household-level variables analysed (tenure, IMD2019 Rank, and household size) for each incentive condition.

Table 2: Sample composition for each incentive condition and the 2021 Census (household-level variables)

Category	Characteristic	£5 incentive	£10 incentive	Census 2021
Tenure	Owns or parts own	73.83%	73.24%	62.3%
Tenure	Does not own or part own	26.17%	26.76%	37.7%
IMD 2019 Rank	1st decile (most deprived)	7.90%	8.13%	10.0%
IMD 2019 Rank	2nd decile	6.79%	8.23%	10.0%
IMD 2019 Rank	3rd decile	7.50%	6.86%	10.0%
IMD 2019 Rank	4th decile	8.51%	8.62%	10.0%
IMD 2019 Rank	5th decile	9.42%	9.50%	10.0%
IMD 2019 Rank	6th decile	10.74%	10.19%	10.0%
IMD 2019 Rank	7th decile	11.14%	10.28%	10.0%
IMD 2019 Rank	8th decile	11.75%	13.03%	10.0%
IMD 2019 Rank	9th decile	12.56%	11.95%	10.0%
IMD 2019 Rank	10th decile (least deprived)	13.68%	13.22%	10.0%
Household size	1	31.10%	29.97%	30.1%
Household size	2	37.28%	38.79%	34.0%
Household size	3	15.91%	11.75%	16.0%
Household size	4	11.14%	13.03%	12.9%
Household size	5 or larger	4.56%	6.46%	6.9%

Table 3: Statistical testing for household-level variables analysed

Category	Χ2	p
Tenure	0.09	0.761
IMD 2019 Rank	3.10	0.960
Household size	11.55	0.021

When considering individual-level variables, education, marital status, and age showed the greatest discrepancies compared to the Census data, as shown in Figure 2 below.

Table 4 further below shows the proportion of individuals in each category of the individual-level variables analysed (education, marital status, age, ethnic group, employment status and sex) for both incentive conditions, as well as the proportions found in the 2021 Census. Below that, Table 5 presents the results of the statistical testing for the variables analysed.

The results show that incentive groups severely overrepresented individuals with Level 4 qualifications and above. Specifically, 48.9% of respondents in the £5 and £10 groups had Level 4 qualifications or above, compared to only 33.9% in the Census data. As is clear from Figure 2, the dissimilarity index score for education was the same for both groups (d of 0.17), indicating a large deviation from the benchmark.

Both groups also overrepresented married individuals and those in civil partnerships, while underrepresenting individuals who have never married. The £10 incentive group performed slightly better in representing marital status, with a lower dissimilarity index score (d of 0.08) compared to the £5 group (d of 0.11).

Regarding age, both groups underrepresented individuals under 40 years old and overrepresented those aged 60 and over. The £5 control group had a more pronounced discrepancy (d of 0.11), whereas the £10 incentive group achieved a dissimilarity index score that indicated good agreement with the benchmark data (d of 0.08).

The ethnic composition of the samples overrepresented ‘white’ individuals, with 84.5% in the £5 group and 86.6% in the £10 group, compared to 81.0% in the Census data. The £5 group slightly outperformed the £10 group in this category with a lower dissimilarity index score (d of 0.03 compared to 0.06). Despite this, both groups demonstrated good agreement with the benchmark data in terms of ethnic group.

Employment status showed very good agreement with the Census data in the £10 group (d of 0.04), but only moderate agreement in the £5 group (d of 0.06). The higher dissimilarity index score in the £5 group indicates that it underrepresented employed individuals to a greater extent while overrepresenting economically inactive individuals more than the £10 group.

Both groups closely matched the Census proportions for country of birth and sex, with dissimilarity indices below 0.05.

Figure 2: Dissimilarity index scores for individual-level variables

Figure 2 shows a bar chart of the dissimilarity index scores for the individual-level variables analysed (education, marital status, age, ethnic group, employment status and sex) for each incentive condition.

Table 4: Sample composition for each incentive condition and the 2021 Census (individual-level variables)

Category	Characteristic	£5 incentive	£10 incentive	Census 2021
Education	Level 4 qualifications or above	48.89%	48.90%	33.9%
Education	Below Level 4 qualifications	51.11%	51.10%	66.1%
Marital status	Never married	28.90%	31.27%	37.9%
Marital status	Married or civil partnership	55.20%	53.20%	44.7%
Marital status	Separated or divorced	15.89%	15.53%	17.4%
Age	0 to 9	9.81%	11.00%	11.3%
Age	10 to 19	11.41%	10.45%	11.7%
Age	20 to 29	6.80%	9.60%	12.6%
Age	30 to 39	11.55%	12.31%	13.7%
Age	40 to 49	13.01%	11.80%	12.7%
Age	50 to 59	12.92%	12.44%	13.6%
Age	60 to 69	14.56%	14.18%	10.7%
Age	70 to 79	12.96%	12.06%	8.6%
Age	80 or over	6.98%	6.16%	4.9%
Ethnic group	White	84.45%	86.62%	81.0%
Ethnic group	Ethnic minorities	15.55%	13.38%	19.0%
Employment status	Employed	53.38%	55.97%	57.4%
Employment status	Unemployed	1.27%	1.28%	3.5%
Employment status	Economically inactive	45.35%	42.75%	39.1%
Country of birth	UK	84.71%	85.90%	82.6%
Country of birth	Non-UK	15.29%	14.10%	17.4%
Sex	Female	50.96%	50.74%	51.0%
Sex	Male	49.04%	49.26%	49.0%

Note: Education, marital status, and employment status information are restricted to individuals aged 16 and over in both the benchmark Census data and the test samples.

Table 5: Statistical testing for individual-level variables analysed

Category	Χ2	p
Education	0.00	0.993
Marital status	2.51	0.285
Age	17.37	0.026
Ethnic group	4.33	0.037
Employment status	2.60	0.273
Country of birth	1.29	0.256
Sex	0.02	0.884

Chapter 4: Summary

The results of this experiment provide limited evidence of an improvement in response rates when increasing the conditional incentive from £5 to £10. Over the three-month period, the £10 incentive group achieved a 1.4 percentage point higher response rate overall compared to the £5 group.

However, the sample composition analysis indicates that the £10 incentive group generally aligned more closely with the population compared to the £5 group, although the differences were small.

Across household-level variables, the £10 group outperformed the £5 group in tenure and IMD2019 Rank, while the £5 group performed slightly better in household size. Both groups overrepresented homeowners and households from the least deprived areas. Household size was relatively well-represented in both groups, with only minor discrepancies observed.

At the individual level, both groups overrepresented married individuals and underrepresented younger age groups. However, the £10 incentive group performed better than the £5 group in these categories. Employment status showed very good agreement with the Census data in the £10 group, but only moderate agreement in the £5 group. Variables such as country of birth and sex showed very good agreement with the Census data for both groups, with only minimal differences between the two incentive conditions.

Overall, these findings suggest that the £10 incentive group more closely resembles the target population across several socio-demographic variables. Offering a larger incentive may help reduce non-response bias and better represent the population. Considering that the current NTS conditional incentive is lower than that of other social surveys, and given the high burden placed on participants who are asked to complete both an interview and a seven-day travel diary involving multiple contacts with interviewers, a £10 incentive might still be worth considering, even if the increase in response rates and representation is modest. However, this would need to be balanced against the significantly higher cost of increasing the incentive.

References

Biemer, P. P., Murphy, J., Zimmer, S., Berry, C., Deng, G., and Lewis, K. (2018). Using bonus monetary incentives to encourage web response in mixed-mode household surveys. Journal of Survey Statistics and Methodology, 6(2), 240–261.

Kaplowitz, M. D., Hadlock, T. D., and Levine, R. (2022). Evaluating the effectiveness of tailored design method principles for internet, phone, mail, and mixed-mode surveys. Social Science Computer Review, 40(3), 499-518.

R Core Team (2024). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag New York.

Wickham, H, François R, Henry L, Müller K, Vaughan D (2023). dplyr: A grammar of data manipulation. R package version 1.1.4.

Wickham H, Hester J, Bryan J (2024). readr: Read rectangular text data. R package version 2.1.5.