Research and analysis

Technical report: Evaluation of the Personalised Support Package (first 18 months)

Published 20 July 2021

Applies to England, Scotland and Wales

1. Introduction

This technical report is accompanying the first report of the Evaluation of the Personal Support Package (PSP), which examines the experiences of support among benefit claimants with health conditions or disabilities in receipt of either Employment and Support Allowance (ESA) or Universal Credit (UC).

The PSP was implemented in 2017 to support people who, owing to illness or disability, are currently unable to work but may be able to in the future. The PSP encompasses a range of existing and new measures and initiatives, designed to offer support that can be tailored to people’s individual needs. These new and existing measures or initiatives include:

• the Health and Work Conversation (HWC)
• a place on the Work Choice (referrals have now come to an end) or Work and Health Programme
• a place on the Journey to Employment Programme (J2E) (only available in certain Jobcentre Districts)
• additional places on the Specialist Employability Support
• additional funding for Access to Work for Mental Health Support Service
• the Small Employer Offer
• a further 300 additional Disability Employment Advisors and Community Partners
• additional support from disability-trained accredited work coaches, who had received specific training around mental health

For the evaluation NatCen carried out two surveys and nine case-studies to explore the implementation of PSP. The surveys were conducted by telephone and the case studies by a mixture of face-to-face group interviews and telephone interviews.

The first survey, undertaken between August and November 2018, was conducted with ESA and UC claimants eligible for PSP support and provided a broad picture of people’s experiences of support provided during their ESA or UC claim. For the qualitative case studies evaluating the PSP, six case studies were conducted between May 2018 and September 2018. Four case studies focused on the implementation and delivery of the J2E initiative. The remaining two explored the wider implementation and delivery of the PSP.

The second survey took place between November and December 2018 and focussed on PSP eligible claimants’ experiences of the Health and Work Conversation (HWC). The purpose of the HWC is to help individuals identify their health, personal and work goals, draw out their strengths, make realistic plans for the future and build their resilience and motivation. For ESA claimants, the HWC techniques are used during a mandatory work focused interview prior to the WCA. For UC claimants on a health journey these can be used throughout the claimant journey, at work coach’s discretion.

Three qualitative case studies were conducted between June and September 2018 involving one-day observations of HWCs and interviews with work coaches and claimants. This report provides technical details of these studies in addition to the information that appears in the main report. The questionnaires, topic guides and recruitment materials sent to respondents have been included in this document as appendices.

The rest of the technical report is grouped into two sections. Section two describes the research into the PSP. It first covers the PSP survey, including the survey’s development, fieldwork details, response rate, sampling, weighting of the data and analysis. It then describes the sampling and response rates, fieldwork details and method of analysis of the qualitative case studies.

Section 3 follows the same format for the research into the HWC. It first describes the method for the HWC survey, including its development, fieldwork, response rate, sampling, the weighting of the data and analysis details, followed by the sampling and response rates, fieldwork details and method of analysis of the HWC case studies.

2. Personal Support Package

2.1 PSP survey

This was the first survey undertaken with ESA and UC claimants who were eligible for PSP. The survey was conducted through computer assisted telephone interviewing (CATI) and took place between August and November 2018. In total, 1,808 individuals took part in the survey.

The questionnaire used for the PSP survey can be found in Appendix 1. The interviews lasted on average 20 minutes. In advance of the survey respondents were sent a leaflet and a letter inviting them to take part, outlining the aims of the survey and letting them know they would be contacted by a telephone interviewer (a copy of these are included in Appendixes 2 and 3). If people did not wish to be contacted there was an opt-out period of two weeks. Fieldwork was conducted on NatCen’s behalf by QRS Market Research.

2.1.1 Questionnaire development

Cognitive interviews were carried out on a number of questions prior to finalising the survey questionnaire. Cognitive interviews aim to investigate how people understand certain questions and how they recall information. The cognitive testing phase, carried out alongside the development of the full survey tool, was used to help develop and test questions before the survey was rolled out into the field. The questions were tested in eight interviewer administered telephone interviews, where the interviewer read out the survey questions and responses before exploring how the respondent answered each question.

The cognitive interviews took place between 3rd and 12th July 2018 and lasted, on average, just over 20 minutes. Participants received a £30 incentive for taking part. Those questions included for testing encompassed: the demographic profile of claimants; attitudes to work; support offered and taken up; reasons for taking up/ not taking up support; what difference support made; and, view on support taken up.

2.1.2 Sampling

The Personal Support Package (PSP) Survey is a single wave survey of claimants who were eligible for PSP support including a sub-group of claimants who had taken up one of the four externally provided PSP strands that were highlighted in the questionnaire, namely J2E, WHP, SES and Work Choice. Throughout this section we will refer to this sub-group as “strand participants”.

Due to the low numbers of strand participants at the time of the survey, all available cases were sampled by DWP. For the rest of the sample of PSP eligible claimants DWP statisticians followed a systematic random sampling strategy, using SAS software and the Proc survey select command. To note, no stratifiers were used by DWP when sampling.

The samples were provided by the DWP statisticians in 2 tranches.

First Tranche Sample

The first sample received from DWP included 15,491 cases. Of these 14,207 were PSP eligible claimants and a further 1,284 strand participants. On receiving the sample key demographic (age, sex, region) variables were checked to ensure there were no missing values. Telephone numbers were also checked at this stage and cases with missing mobile and home phone numbers were excluded. Nineteen cases were excluded at this stage due to missing phone numbers.

Cognitive Sample

The next stage was to draw a cognitive sub-sample. A random sample of 181 cases, of which 75 PSP were strand participants.

Extra J2E Sample

From the remaining sample, 132 more cases were allocated to J2E. Of these, 18 were strand participants.

Extra Qualitative Sample

Lastly, 240 more cases were allocated to an extra qualitative sample. Of these 198 42 were strand participants.

The remaining 14,919 cases after all the above exclusions, shown in Table 1, were assigned to the final quantitative sample.

Table 1: Remaining first tranche sample after exclusions and further sampling

	Non strand participants	Strand participants	Total
Total sample received from DWP	14,207	1,284	15,491
Missing telephone numbers	17	2	19
Cognitive sub-sample	106	75	181
Extra J2E sub-sample	114	18	132
Extra qualitative sub-sample	198	42	240
Remaining sample	13,772	1,147	14,919

Second Tranche Sample

The second sample received from DWP included 11,000 cases. Of these 675 were strand participants. On receiving the sample key demographic variables (age, sex, region) were checked to ensure there were no missing values. Telephone numbers were also checked at this stage and cases with missing mobile and home numbers were excluded. Thirteen non-strand participants cases were excluded at this stage due to missing phone numbers.

The final sample file after sampling related exclusions consisted of 25,906 cases across both strand participants and non-strand participants (14,919+10,987). Further exclusions due to duplicate contacts and wrong telephone numbers were carried out by the research team, leading to a final issued sample of 24,141 cases: 22,440 non-strand participants and 1,701 strand participants.

2.1.3 Weighting

The profile of achieved interviews was compared to the available population demographics from DWP to determine whether weighting would be required. The comparison showed significant difference on the profile by region, age, sex and disability type. This is due to deduplication, missing contact information and different levels of non-response across the groups. Hence, a decision was made to use calibration weighting to align the sample profile to the eligible population profile provided by DWP. The two survey groups, non-strand participants and strand participants were calibrated separately, to their respective population profile.

The survey data for both strand participants and non-strand participants were weighted to the marginal age, sex, region and disability type. However, because of much smaller sample sizes, the strand participant data for age bands and region were recoded into fewer groups to ensure sample sizes in each group were sufficiently high. After calibration the weighted data for non-strand participants and strand participants should exactly match the population across these four dimensions. This is shown in Tables 2 and 3 below.

Table 2: Weighted and unweighted non-strand participant sample distribution by age, sex, region and disability type

Region	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
North East	7.8	6.0	7.8
North West	7.7	11.5	7.7
Yorkshire and Humber	9.6	8.8	9.6
East Midlands	7.1	9.6	7.1
West Midlands	14.4	7.7	14.4
East of England	10.3	8.7	10.3
London	7.5	10.3	7.5
South East	6.5	11.4	6.5
South West	10.9	9.6	10.9
Wales	9.7	7.3	9.7
Scotland	8.4	9.1	8.4

Age	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
16–20	2.2	3.1	2.2
21–30	19.4	18.9	19.4
31–40	20.9	18.0	20.9
41–50	24.1	26.6	24.1
51–60	24.9	27.1	25.0
61–70	8.4	6.2	8.4

Gender	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
Female	48.9	46.7	48.9
Male	51.1	53.3	51.1

Disability type	Population	Unweighted respondents	Respondent weighted by final weight
Diseases of the Circulatory System	2.9	4.0	2.9
Diseases of the Musculoskeletal System and Connective Tissue	9.1	9.4	9.1
Diseases of the Nervous System	4.3	5.1	4.3
Mental and Behavioural Disorders	55.4	52.1	55.4
Other[footnote 1]	28.3	29.4	28.3

Base:
Population: 166,229
Unweighted respondents: 1,633

The design effect^{[footnote 2]} for the non-strand participant sample is estimated at 1.16, which means the effective sample size^{[footnote 3]} for the analysis of the total sample is c. 1,409.

Table 3: Weighted and unweighted strand participant sample distribution by age, sex, region and disability type

Region	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
The North	18.5	14.3	18.5
Yorkshire and Humber	16.0	14.9	16.0
Midlands	21.3	23.4	21.3
East of England	7.6	8.0	7.6
London	6.5	11.4	6.5
Wales and the South	19.9	16.0	19.9
Scotland	10.2	12.0	10.2

Age	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
16–30	28.1	23.4	28.1
31–50	46.1	42.9	46.1
51–70	25.8	33.7	25.8

Gender	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
Female	43.5	44.6	43.5
Male	56.5	55.4	56.5

Disability type	Population (%)	Unweighted respondents (%)	Respondent weighted by final weight (%)
Diseases of the Circulatory System	2.4	3.4	2.4
Diseases of the Musculoskeletal System and Connective Tissue	9.6	13.1	9.6
Diseases of the Nervous System	4.7	5.1	4.7
Mental and Behavioural Disorders	53.6	48.0	53.6
Other[footnote 4]	29.7	30.3	29.7

Base:
Population: 1,821
Unweighted respondents: 175

The design effect for the strand participant sample is estimated at 1.09, which means the effective sample size for the analysis of the total sample is c. 161.

Finally, a combined weight was produced to enable the analysis of both non-strand participants and strand participants together on key estimates. This entailed stacking the non-strand participant and strand participant weights on top of each other to create one weight variable covering all achieved cases. The design effect for the overall sample is estimated at 1.26, which means the effective sample size for the analysis of the total sample is c. 1,438.

2.1.4 Response rate

The sample issued for the PSP survey included 24,141 claimants. Of the initial sample 2,590 did not have valid phone numbers and were excluded from the final sample size used to calculate response rates. When fieldwork started some respondents were also found not to be suitable for the survey because they had not been to a Jobcentre and therefore had no experience of any of the types of support provided there. A screener question was introduced to screen out those people who had not been to a Jobcentre. As a result of this 738 people were screened out of the survey. This included 237 interviews conducted before the screener was introduced with respondents who had never had a face-to-face meeting with a Jobcentre. As a result, these respondents could not fully respond to the survey and they were not included in the final number of interviews. Following discussions with DWP the target number of interviews was revised from 2,000 to 1,800. The final sample size was therefore 20,813, from which a response rate of 9% was achieved, resulting in 1,808 interviews.

2.1.5 Sub-group analysis

The quantitative findings presented in the report are based on frequencies and, where sample size allowed, cross-tabulations of questions to explore the differences between various sub-groups. These included how responses differed by a range of socio-demographic factors such as age, gender and education, as well as attitudinal variables collected in the survey on self-reported health status and attitudes towards work. All percentages cited in the report are based on weighted data and are rounded to the nearest whole number. Don’t know and refusal responses were not included in questions’ base sizes.

Logistic regression was used to test whether differences were statistically significant and only results significant at the 95% level or above were reported. This means that the probability of having found a difference of at least this size, if there was no actual difference in the population, is 5% or less. Regression analysis aims to summarise the relationship between a ‘dependent’ variable and one or more ‘independent’ variables. It is often undertaken to support a claim that the phenomena measured by the independent variables cause the phenomenon measured by the dependent variable. However, the causal ordering, if any, between the variables cannot be verified or falsified by the technique. Logistic regression was used in this case because the dependent variables were categorical and strictly speaking linear regression assumes both independent variables (for example, demographic information such as age) and dependent variables (for example, attitudinal questions) are measured on an interval-level scale.

2.2 Personal Support Package qualitative case studies

Six case studies were conducted between May 2018 and September 2018. Four case studies focused on the implementation and delivery of the J2E initiative. The remaining two explored the wider implementation and delivery of the PSP. Case studies were selected to ensure the overall sample included a mix of areas where UC had been rolled out and places where claimants were receiving ESA.

Each case study comprised of three-way telephone or face-to-face interviews with work coaches, team leaders and disability employment advisers (DEAs). Individual interviews were completed, where group interviews were not possible, due to time constraints. Individual telephone interviews were conducted with Community Partners and Small Employer Advisors and external providers.

Sampling and recruitment of Jobcentre Plus staff

Across the six case studies a total of 63 interviews were conducted with work coaches, work coach team leaders (team leader), DEAs, Community Partners, Small Employer Advisors and external providers.

Table 4: JCP pilot staff achieved sample

Role	District 1	District 2	District 3	District 4	District 5	District 6
Work coach	2	3	2	2	4	4
Team leader	3	3	3	3	2	2
DEA	1	3	0	3	3	2
Community Partner	1	1	2	1	1	2
Small Employer Advisor	0	0	0	1	0	0
Provider	2	2	1	2	1	1
Total	9	12	8	12	11	11

DWP issued a sample of Jobcentre Plus staff in each case study area. Staff were randomly selected and sent an email with an attached information sheet inviting them to participate in a telephone interview, the invitation explained the voluntary nature of participation.

Distinct topic guides were developed for use in interviews across the two types of case studies and with different staff roles. Group interviews took up to 90 minutes and individual interviews took up to 60 minutes.

Sampling and recruitment of providers in J2E case study areas

DWP issued a sample of J2E providers operating in the selected case study areas. Providers were approached via email, with an attached information sheet that explained the voluntary nature of participation. All providers sent an invitation to participate in the evaluation took part in a telephone interview. Interviews lasted between 60 and 90 minutes.

In the two case study areas that focused on PSP more broadly providers were selected via a snowball approach. Jobcentre Plus staff were asked to identify PSP or existing local providers they had referred eligible PSP claimants to on a regular basis. With permission Jobcentre Plus staff were able to share the contact details of providers who were recruited via email. Interviews were conducted via the telephone and lasted between 30 and 60 minutes.

Sampling and recruitment of claimants

In the four case studies that focused on J2E a total of 13 claimants took part in a face to face or telephone interview. In the two case studies that focused on the implementation and delivery of PSP more generally a total of 11 telephone interviews were conducted. The achieved sample across the two different types of case studies included variation on gender, benefit claimed, length of current claim, health condition and age. A full breakdown of achieved sample by characteristics can be found in Table 5.

Table 5: Achieved sample for claimant interviews the PSP case study areas

Gender	J2E case studies	PSP case studies
Male	10	6
Female	3	5

Benefit claimed	J2E case studies	PSP case studies
ESA	13	11
UC	0	0

Length of current claim	J2E case studies	PSP case studies
<2 years	7	1
2 - 5 years	5	8
>5 years	1	2

Health condition	J2E case studies	PSP case studies
Mental	6	7
Physical	1	0
Both	5	4
Other	1	0

Age	J2E case studies	PSP case studies
18-29	2	1
30-49	5	9
50+	6	1

Total:
J2E case studies: 13
PSP case studies: 11

DWP issued a sample of PSP eligible participants in each of the case study areas. All claimants in the sample were sent an advance letter informing them of the nature of participation and offering them the opportunity to opt-out of being contacted. Claimants who did not opt-out were called and recruited to the sample. For the J2E case studies, a number of screening questions were asked at recruitment to ensure claimants had experience of taking part in the programme or could recall being offered and declining participation. In the two case studies focusing on PSP more broadly, screening questions were asked to establish whether claimants had had a meeting with their work coach within the past year, indicating that they would have been offered a PSP initiative. If claimants were eligible for participation and consented to take part they were recruited to the sample. A confirmation letter, SMS or email was sent a day before the interview to those claimants who agreed to take part. Claimants were contacted up to five times in total. Interview length ranged between 30 and 60 minutes and were conducted either via the telephone of face-to-face.

With participants’ permission, interviews were audio recorded. Recordings were transcribed, and the transcripts analysed thematically.

3. Health and Work Conversation

3.1 HWC survey

A telephone survey lasting 20 minutes was undertaken with ESA and UC claimants who were eligible for HWC. The survey took place between November and December 2018. In total, 1,006 individuals took part in the survey; 506 UC claimants and 500 ESA claimants.

The questionnaire used for the HWC survey can be found in Appendix 4. In advance of the survey respondents were sent a leaflet and a letter inviting them to take part, outlining the aims of the survey and letting them know they would be contacted by a telephone interviewer (a copy of these are included in Appendix 5 and 6). If people did not wish to be contacted there was an opt-out period of one week. Fieldwork was conducted by NatCen’s telephone unit.

Pilot interviews were conducted to test the questionnaire for any issues with understanding. For the pilot 30 interviews took place between the 10th and 20th of September 2018. There was a roughly even split between UC (16) and ESA claimants (14). Interviews were conducted by telephone to match the mode of the main survey and lasted, on average, 20 minutes. A number of questions were focused on in the pilot including: participant demographic characteristics; experience of the HWC; setting goals; follow-up meetings and perceived outcomes of the HWC.

3.1.1 Sampling

The Health and Work Conversation (HWC) Survey was designed as a survey to gauge claimant experiences and to evaluate the effectiveness of the initiative. This was carried out with respect to claimants receiving either Universal Credit (UC) or Employment Support Allowance (ESA).

Whereas a sample of the population of UC claimants was drawn by DWP statisticians, only ESA claimants who had been recorded as having had an HWC in the previous 4-6 weeks were included in the sample. In effect, the ESA sample was a census.

The samples were provided by DWP in three tranches – each of which is broken down in detail below. The sampling for UC cases was carried out by DWP statisticians and followed a simple random sampling strategy, using SAS Proc survey select (method: systematic random sampling). To note, no stratifiers were used by DWP when sampling.

First Tranche Sample

The first sample received from DWP included 21,373 cases. Of these 19,916 were UC claimants and 1,457 were ESA claimants. On receiving the sample, some key demographic variables (age, sex, region) were checked to ensure there were no missing values. Cases without a valid telephone number (neither landline nor mobile) were then excluded from the sample at this stage. Of the UC claimants, 53 cases were excluded by these criteria, compared to two ESA claimants.

After these exclusions were made, all the remaining ESA population (1,455 cases) was issued. Of the UC sample, a further sub-sample of 9,000 cases was drawn. To do so, a stratified random sample was taken using a simple systematic approach. Within this, age group, sex and region were used to stratify the sample.

This gave a total sample of 9,000 UC cases and 1,455 ESA cases to be issued in the first tranche of fieldwork. Once the UC sample had been drawn, 10,863 UC cases were left. These were held in reserve for future sample if a boost were required to achieve a sufficient survey response.

Table 6: Remaining first tranche sample after exclusions and further sampling

	UC	ESA	Total
Sample received from DWP	19,916	1,457	21,373
Missing telephone numbers	53	2	55
Sampled	9,000	1,455	10,455
Remaining	10,863	0	10,863

Second Tranche Sample

The second sample received from DWP contained 387 ESA cases. Taking the same steps as before, one case was excluded due to there being no valid phone number associated with that case. Furthermore, checks were carried out to ensure that no duplicate cases were present.

In addition to the 386 valid ESA cases which were issued in the second tranche of fieldwork, a further 3,000 UC cases were drawn from the reserve of cases left after the first tranche. These cases were sampled using systematic random sampling, with age group, sex and region as stratification variables.

Table 7: Remaining second tranche sample after exclusions and further sampling

	UC	ESA	Total
Remaining from first tranche	10,863	0	10,863
Additional sample from DWP	0	387	387
Missing telephone numbers	0	1	1
Sampled	3,000	386	3,386
Remaining	7,863	0	7,863

Third Tranche Sample

The final sample of claimants received from DWP contained 450 ESA cases. Following the same procedure described above, there was a single case which had neither a valid landline nor mobile number. Once this case had been removed, this left 449 ESA cases in the third tranche sample.

Considering all three tranches of sampling, Table 8 (below) breaks down the sample by tranche and benefit type. All in all, 14,290 cases were sampled, of which 2,290 were ESA claimants. Meanwhile, a total of 57 cases were excluded on the basis that there was no valid phone number associated.

Table 8: Total sample, tranches 1-3

Sample received from DWP

Tranche	UC	ESA	Total
1	19,916	1,457	21,373
2	-	387	387
3	-	450	450

Missing telephone numbers

Tranche	UC	ESA	Total
1	53	2	55
2	-	1	1
3	-	1	1

Sample

Tranche	UC	ESA	Total
1	9,000	1,455	10,455
2	3,000	386	3,386
3	0	449	449

Totals

	UC	ESA	Total
Total sampled	12,000	2,290	14,290
Remaining (not sampled)	7,863	-	7,863

3.1.2 Weighting

Once fieldwork was complete, weights were produced to account for bias arising from non-response. The profile of achieved interviews was compared to the demographic profile of the ESA and UC population – provided by DWP – to determine whether weighting would be required. The comparison showed significant difference on the profile by region, age and sex. This is due to deduplication, missing contact information and different levels of non-response across the groups. As a result, a decision was taken to use calibration weighting to align the sample profile to the eligible population profile. The two survey groups, ESA and UC, were calibrated separately, to their respective population profiles. Furthermore, the weights were also scaled to the counts for the respective claimant populations.

The survey data for ESA and UC claimants were weighted to the marginal age, sex, and region. Due to differences in the availability and rollout of the two benefit types, region was recoded in each case to ensure that sample sizes in each group were sufficiently high. When the calibration weights are applied, the breakdown of age, sex and region for the population matches the weighted breakdown of participants. This is demonstrated in Tables 9 and 10 below.

Employment Support Allowance Weighting

As has been outlined, the survey of ESA claimants was in effect a census. The population, in this case, was composed of all claimants who had participated in an HWC within 4-6 weeks of the sample being drawn. Once those with neither a valid landline nor mobile number had been excluded, the demographic profile of this population was taken. This was then referred to in the calibration.

Table 9: Weighted and unweighted ESA sample distribution by age group, sex and region

Region	Population %	Unweighted respondents %	Weighted respondents %
North East and Yorkshire	8.8	7.9	8.8
North West	16.3	15.4	16.3
Midlands	11.4	11.6	11.4
South England	5.4	7.5	5.4
East England	9.5	9.9	9.5
London	16.8	19.7	16.8
Wales	12.4	11.8	12.4
Scotland	19.3	16.2	19.3

Age	Population %	Unweighted respondents %	Weighted respondents %
16-30	26.3	22.1	26.3
31-40	22.6	18.9	22.6
41-50	19.6	17.0	19.6
51-60	22.9	29.0	22.9
61+	8.7	13.0	8.7

Gender	Population %	Unweighted respondents %	Weighted respondents %
Male	50.5	50.5	50.5
Female	49.5	49.5	49.5

Base:
Population: 2,285^{[footnote 5]}
Unweighted respondents: 507
Weighted respondents: 2,285

The design effect^{[footnote 6]} for the ESA sample is estimated at 1.07, which implies an effective sample size^{[footnote 7]} of 475.

Universal Credit Weighting

The weighting of UC claimants was based on the demographic profiles of the UC population (1,059,401 claimants) provided by DWP. Following the same steps as for ESA claimants, calibration weights for UC were produced.

Table 10: Weighted and unweighted UC sample distribution by age, sex and region

Region	Population %	Unweighted respondents %	Weighted respondents %
North East and Yorkshire	14.7	14.4	14.7
North West	16.6	12.8	16.6
Midlands	15.0	16.8	14.9
Wales and South West England	16.0	16.6	15.9
East England	7.2	9.4	7.2
London	13.8	12.6	13.8
South East	7.1	9.6	7.1
Scotland	9.7	7.8	9.7

Age	Population %	Unweighted respondents %	Weighted respondents %
16-30	38.3	24.6	38.3
31-40	24.7	18.6	24.7
41-50	17.9	20.0	17.9
51-60	14.7	28.8	14.7
61+	4.4	8.0	4.4

Gender	Population %	Unweighted respondents %	Weighted respondents %
Male	47.5	49.8	47.5
Female	52.5	50.2	52.5

Base:
Population: 1,059,401
Unweighted respondents: 500
Weighted respondents: 1,059,401

The design effect for the UC sample is estimated at 1.24, which implies an effective sample size of 405.

Combined weights

Once the calibration weights for the individual benefit types had been computed, a combined weight was produced to enable the analysis of both ESA and UC claimants together on key estimates. This entailed stacking the ESA and UC weights on top of each other to create one weight variable covering all achieved cases, which was calibrated to the combined population profile of ESA and UC claimants. The design effect for the overall sample is estimated at 2.48, which implies an effective sample size for the total sample of 407.

3.1.3 Response rate

The issued sample included 14,290 cases (12,000 UC and 2,290 ESA). The sample used by the telephone interviewers before the target number of interviews was reached was 9,357, of which 1,790 were ineligible due to incorrect telephone numbers.

Two screener questions were also introduced to the survey to ensure participants had attended meetings at the Jobcentre since the start of their ESA or UC claim and that they had a health condition. Participants who had not been to a Jobcentre were not asked to complete the survey, as they would not have had an opportunity to take part in either the HWC in ESA areas or participate in any of the HWC techniques in UC areas. Those without a health condition were also routed out of the survey, as the sample was intended to include only those with some kind of health condition. Together this screened out 137 ESA and 619 UC claimants from participating in the survey.

The total eligible sample used was therefore 7,567 from which 1,006 interviews were achieved, resulting in a response rate of 13%.

3.1.4 Sub-group analysis

Logistic regression was used to test whether differences were statistically significant and only results significant at the 95% level or above were reported. This means that the probability of having found a difference of at least this size, if there was no actual difference in the population, is 5% or less. Regression analysis aims to summarise the relationship between a ‘dependent’ variable and one or more ‘independent’ variables. It is often undertaken to support a claim that the phenomena measured by the independent variables cause the phenomenon measured by the dependent variable. However, the causal ordering, if any, between the variables cannot be verified or falsified by the technique. Logistic regression was used in this case because the dependent variables were categorical and strictly speaking linear regression assumes both independent and dependent variables are measured on an interval-level scale.

3.2 HWC qualitative case studies

Three HWC qualitative case studies were completed between June and September 2018. Case studies were chosen to ensure the achieved sample included a mix of Jobcentre Plus staff with experience of conducting HWC techniques in a ESA and UC setting, as well as claimants on ESA or on a health journey as part of their UC claim. Case study areas were also selected to ensure the three geographical locations chosen covered a mix of rural and urban areas.

Each case study comprised of observations of the HWC with ESA claimants or the First Commitment meeting with UC claimants and in-depth interviews with Jobcentre Plus staff. In-depth interviews were also conducted with ESA claimants and UC claimants outside of the case study areas. A wider sample than just claimants within the Jobcentre Districts selected as case studies was issued to ensure the target number of interviews were achieved.

Health and Work Conversation and First Commitment meeting observations

Between two and three observations of Jobcentre meetings were carried out in each of the three case study areas. Observations were arranged via a single point of contact issued by DWP. Researchers gained verbal consent from Jobcentre Plus staff and the ESA / UC claimant participating in the HWC or First Commitment meeting to observe. An observation pro-forma was developed, and notes were taken during each observation. These notes were used to draft the topic guides used for the Jobcentre Plus staff and ESA or UC claimant interviews. The notes were also thematically organised and used in the analysis alongside data from claimant and staff interviews.

Qualitative interviews with Jobcentre Plus staff

Across the three case studies a total of 27 in-depth telephone interviews were conducted with work coaches, work coach team leaders (team leaders) and disability employment advisors (DEA).

Table 11: JCP pilot staff achieved sample for HWC case studies

Role	District 1	District 2	District 3	Total
Work coach	5	6	5	16
Team leader	2	1	2	5
DEA	3	2	1	6
Total	10	9	8	27

DWP issued a sample of Jobcentre Plus staff in each case study area. Staff were randomly selected and sent an email with an attached information sheet inviting them to participate in a telephone interview. Distinct topic guides were developed for use in ESA and UC case study areas. This was due to staff using the techniques at different stages of a claimant journey. For example, in the ESA context all HWC techniques should be used during one work-focused interview. In comparison, UC work coaches have the flexibility to use techniques as and when they feel it is appropriate for a UC claimant on a health journey.

Telephone interviews were arranged with staff members who agreed to participate. Interviews lasted between 30 minutes to one hour.

Qualitative interviews with ESA and UC claimants

A total of 24 in-depth face to face interviews were conducted with ESA claimants and UC claimants who had some experience of HWC techniques. The sample included variation on type of benefit claimed, gender, age, health condition. Table 12 shows the full breakdown of the achieved sample by characteristics.

Table 12: Achieved participant sample for HWC case studies

Gender	Total
Male	13
Female	11

Benefit claimed	Total
ESA	13
UC	11

Health condition	Total
Mental	15
Physical	7
Both	2

Age	Total
18-29	3
30-49	11
50+	10

Total: 24

The sample was drawn from a sample frame of ESA claimants who had recently taken part in a HWC. The UC sample drawn from claimants who had taken part in their first commitment meeting was issued by DWP. All claimants were sent an advance letter informing them of the voluntary nature of participation and offering them the opportunity to opt-out of being contacted. Claimants who did not opt-out were called and recruited to the sample.

A number of screening questions were asked at recruitment to ensure claimants had experience of one or more HWC techniques. Claimants were screened out of the study if they had no experience of the techniques. A confirmation letter, SMS or email was sent a day before the interview to those claimants who agreed to take part. A topic guide was designed in collaboration with DWP. The topic guide captured claimants’ recall and views and experiences of the different HWC techniques. It also captured claimants’ views and pursuit of goals or actions as a result of engaging in HWC techniques. Interviews lasted between 30 to 45 minutes and were conducted either via the telephone or face-to-face. Proxy interviews were conducted where a participant’s health or disability stopped them from being able to participate.

All Jobcentre Plus staff and claimant interviews were audio recorded, with permission obtained from all participants. Recordings were transcribed, and the transcripts analysed thematically.

1. Other includes Injury and Poisoning. These were grouped together because of small sample sizes. ↩

2. The design effect, often called just deff, quantifies the extent to which the expected sampling error in a survey departs from the sampling error that can be expected under simple random sampling. The design effect increases for more complex sample designs and when weighting adjustments are applied to the final results of the survey. ↩

3. The effective sample size is an estimate of the sample size that a survey conducted using simple random sampling would have required to achieve the same sampling error as computed in the study that did not employ simple random sampling. The effective sample size is computed by dividing the sample size by the design effect (deff). ↩

4. Other includes Injury and Poisoning. These were grouped together because of small sample sizes. ↩

5. Of the issued sample, 2290 cases were classified as ESA claimants at the point of issue. When asked about the current benefits that they receive, a small number of ESA and UC claimants responded that they received a different benefit than that which had been recorded. In these instances, the claimant response was deemed to supersede the administrative data. This led to a net movement of five cases from ESA to UC. The ESA population therefore declined from 2290 to 2285 cases. ↩

6. The design effect – often referred to as DEFF – quantifies the extent to which the expected sampling error in a survey departs from the sampling error that can be expected under simple random sampling. The design effect increases for more complex sample designs and when weighting adjustments are applied to the final results of the survey. ↩

7. The effective sample size is an estimate of the sample size that a survey conducted using simple random sampling would have required to achieve the same sampling error as computed in the study that did not employ simple random sampling. The effective sample size is computed by dividing the sample size by the design effect (DEFF). ↩