Independent report

Encouraging employers to advertise jobs as flexible

Published 5 March 2021

1. Executive summary

1.1 Background

The Government Equalities Office established the Gender and Behavioural Insights (GABI) Programme in partnership with the Behavioural Insights Team (BIT) in 2017. The GABI programme aims to build evidence on what works to improve gender equality in the workplace. As part of this programme, BIT partnered with Indeed, a major UK job site (Indeed), and Harvard Business School, to test a behaviourally-informed intervention to encourage employers to advertise more jobs with flexible working options.^{[footnote 1]}

According to research by Timewise, 93% of non-workers^{[footnote 2]} who would like to work prefer flexibility, while only 22% of ‘quality jobs’^{[footnote 3]} are advertised as flexible. The outbreak of COVID-19 pandemic, ensuing lockdown and increased remote working have only strengthened these preferences, with almost all employees desiring greater flexibility. Once in the job, 60% of workers end up working flexibly. As women are twice as likely to work flexibly, this lack of transparency is likely to affect them more. Additionally, women may be particularly averse to ambiguity in job adverts^{[footnote 4]} and may avoid specifically asking for flexibility due to concerns about negative employer reactions. This is an issue not just for women, as research by home, working remotely) and other arrangements (such as job sharing). the Department for Business, Energy and Industrial Strategy (BEIS) suggests that fathers can be twice as likely to have their flexible working request rejected.

1.2 Trials, interventions and methodology

In 2019, we ran a series of 3 randomised controlled trials (RCTs) to test whether changes to the choice architecture of job advert templates can encourage employers to advertise more jobs with flexible working options, and the impact on job seekers.

Indeed: Round 1

Between April to May 2019, we conducted a first large field RCT with the jobsite Indeed (N=55,744 advertisers), testing the impact of introducing a prompt in the job listing template which gave employers the option to advertise jobs with a choice of flexible working options, compared to business-as-usual with no such prompt (Section 1, ‘Round 1’).

Indeed: Round 2

Between September and December 2019, we attempted a replication in a second large field RCT with Indeed (N=91,309 advertisers), where we aimed to test the same prompt and 2 arms with additional behaviourally-informed messaging (highlighting that flexibility is a legal right in the UK, and that flexibility is gender inclusive, given both men and women desire it), compared to business-as-usual (Section 2, ‘Round 2’).

Across these 2 trials, our primary outcome measure was whether or not the resultant job posting mentioned flexible working options. We compared postings which had been subject to the prompt with a control group of postings which had not. We used web scraping to establish the proportion of job advert postings that offered flexible working options across both the treatment and control groups. Our secondary outcome measure was the number of applications received within 2 weeks after the job posting, to determine whether flexible jobs attract more applicants.

In total, these field trials involved almost 100,000 employers posting more than 780,000 job adverts that elicited over 19 million applications.

Online trial

Finally, in November 2019, we conducted an online experiment (N=5,034) on BIT’s online experimentation platform Predictiv to explore any gender differences in applicants’ propensity to shortlist jobs mentioning flexibility, compared to job adverts without such mentions (Section 3, ‘Online trial’). We varied how specific the description of flexibility was, testing 4 treatment arms compared to the control. This trial was intended to complement the 2 RCTs with Indeed, as Indeed was unable to provide data on gender, so we could not explore any potential differential effect on women and men.

1.3 Findings

Round 1

In Round 1, we found that employers exposed to the prompted choice page in the job listing template were on average 20% more likely to advertise their job with flexible working options (an increase of 7 percentage points, p<0.001), compared to the control group where on average 35% of job adverts offered flexibility.

Exploratory analysis suggests that this effect was mostly driven by the increased offer of Flexitime^{[footnote 5]} but all types of flexible working showed a significant increase. Looking at jobseeker response, we found that job adverts offering flexible working attracted 30% more applicants (p<0.05), though this is likely an overestimate due to potential spillover effects between treatment and control groups.

Round 2

Due to severe implementation challenges, randomisation of the trial was compromised which means our findings do not withstand the highest robustness standards of RCTs. Initially no control group was included and employers treated in Round 1 were included in this trial, which was not in line with the intended research design. Whilst we have secured data on never-exposed employers to serve as a control group, our balance checks have identified notable imbalances. These randomisation issues suggest that Round 2 can be only interpreted as a weakly matched quasi-experimental design.

Subject to these caveats, the indicative findings broadly replicate Round 1, albeit with small differences in effect sizes. Employers exposed to prompted choice were still more likely to advertise jobs as flexible (6 percentage points) and such offers received more applications (19 percent). However, additional alternative messaging on the prompted choice page was not more effective than the effect of prompted choice.

Online trial

In the online trial, we found that job adverts featuring specific mentions of flexibility were equally more likely to be shortlisted by both women and men, than the control full-time offer. Any mention of flexible working increased the likelihood women would shortlist the job, though specific descriptions were preferred. For men, only specific descriptions increased preferences to shortlist the job compared to no mention of flexible working.

1.4 Implications

These large-scale trials show strong results suggesting small changes to the choice architecture of job postings can encourage employers to advertise more jobs with flexible working options, subject to some implementation challenges and methodological limitations. Furthermore, jobs advertised in this way tend to attract more jobseekers, possibly with close to an equal gender split between men and women.

Based on these findings, we recommend implementation and scaling of prompted choice to encourage employers to advertise more jobs as flexible. We estimate that a roll-out of this intervention across Indeed alone could result in around 174,000 more jobs per year with flexible working options.

Given this research has been completed before the COVID-19 pandemic, this estimation does not include considerations of the broader shifts in the labour market towards flexibility and the likely positive interaction between these trends and our intervention.

1.5 Research context

Flexible working can be key to enabling people with caring responsibilities to reconcile the competing demands of work and care. All employees in the UK have the legal right to request flexible working arrangements, though there is no onus on employers to offer them or to be transparent about what they may be willing to offer.

Women provide twice as much childcare as men and are twice as likely to work flexibly. Boosting the supply of flexible jobs is therefore key to expanding the pool of jobs available for people with caring responsibilities, which we expect disproportionately to benefit women at the current time. Making flexible working more widely available also has the potential to normalise flexible working for both women and men.

Flexibility is hard to come by

However, whilst 93% of non-workers who would like to work prefer flexibility, they can struggle to find jobs advertised as such. For instance, research from Timewise found that only 22% of ‘quality jobs’ (which they define as permanent and paying £20,000 or more per year) are advertised as flexible. This demand-supply gap is aggravated by the lack of transparency about potential flexible working options, where potentially flexible jobs are not advertised as such. This means that people either cannot find suitable job vacancies or they have to actively request flexibility. Research suggests that this ambiguity about flexibility can be particularly discouraging for women,^{[footnote 6]} who may be more likely to be averse to poor clarity. Research also indicates that 2 in 5 women will avoid bringing up flexibility because they fear the negative impact on their chances of being hired.

Changing ‘choice architecture’ of jobs adverts

One promising avenue for a behavioural intervention is to improve the offer of flexibility on job postings on third-party job sites. This is because jobsites enable access to a large pool of employers and jobseekers at a point in time when they are about to put out a job posting or apply, respectively.

The design and presentation of choices can disproportionately affect the decisions we make.^{[footnote 7]} People frequently make choices using intuitive ‘fast’ thinking that relies on simple cues from the environment, instead of using systematic slow deliberation.^{[footnote 8]} This is why we need to think carefully about ‘choice architecture’: which options are available during a given decision point, how they are framed, and what happens if people fail to make a deliberate choice?

In the context of online job advertising, the job posting template is an example of choice architecture. The job site which we partnered with as part of this trial provides employers with a job listing template. The original template did not include a clear flexible working category for employers to use to advertise jobs’ flexible working options. This means that the only way for employers to inform jobseekers that a job is suitable for flexible working is to choose to mention it in the text of the advert itself.

However, most employers using the job site do not do so (65%).^{[footnote 9]} There is thought to be a gap here between what is being advertised and what is truly on offer. The evidence shows that the majority of jobs ultimately include an element of flexibility, as 63% of UK full-time employees work flexibly.

Reasons behind the lack of flexibility and how to address them

Behavioural theory may help to explain why employers do not advertise jobs as flexible, when they could be willing to offer flexibility. Reasons could include status quo bias^{[footnote 10]} which favours full-time work, and ambiguity aversion^{[footnote 11]} which may discourage consideration of a range of flexible working patterns.

Debiasing the choice environment is key to improving equality in the workplace. Prompting choice is a promising behavioural solution to counter the lack of transparency of job flexibility. Such prompts can encourage people to reflect on their preferences, reveal them, and encourage them to select the socially desirable option.^{[footnote 12]} By adding a prompt to employers asking them to clearly indicate whether or not a job can be done flexibly, we can remove the ease of inaction and prompt them to make a deliberate decision on the type of job they are offering. This can help to both reduce the lack of transparency in the working options being advertised to job seekers in individual job adverts, and also potentially encourage employers to offer more flexible jobs overall.

Evidence shows changes to job adverts change behaviour

Behaviourally-informed changes to job adverts can influence jobseeker behaviour. For instance, one US study found that simply adding a single sentence about how many people applied for the job can increase women’s application rates.^{[footnote 13]} Another study has shown that mentioning that a salary is negotiable in the job advert increased women’s propensity to negotiate.^{[footnote 14]} Most relevant of all, a recent experiment in China has found that the unsolicited offer of roles advertised with flexibility options attracted more applicants who were married women and, to a lesser extent, married men.^{[footnote 15]} Given this research and women’s greater tendency to work flexibly, we hypothesised that mentioning flexibility in job adverts should encourage more applicants - and in particular more women - to apply.

2. Round 1

2.1 Background

In 2017, the Government Equalities Office (GEO) commissioned the Behavioural Insights Team (BIT) to deliver a 3-year programme of work - the Gender and Behavioural Insights (GABI) programme. GABI aims to build evidence on what works to improve gender equality in the workplace, by using behavioural insights and empirical approaches. The programme includes the running of trials to design and test interventions to improve gender equality in the UK.

As a part of the GABI programme, BIT partnered with a major UK job site to run a large field trial. We designed an intervention informed by behavioural insights which appeared within the job site’s job listing template used to write and post job offers. The intervention aimed to encourage more employers to clearly advertise the flexible working options they could support in relation to the job on offer. Between April and May 2019, we helped implement and rigorously evaluate this intervention. This report notes findings from this trial.

We would like to thank Professor Iris Bohnet (Harvard Kennedy School), Associate Professor Mike Luca (Harvard Business School), and PhD candidate Heidi Liu (Harvard Kennedy School) for providing expert advice on the intervention design, and to PhD candidates Jeff Fosset (Harvard Business School) and Stephanie Chan- Ahuja (London Business School) for their valuable comments at the analysis stage.

2.2 Intervention design and test methodology

We partnered with Indeed and Harvard Business School, to conduct a large field randomised controlled trial between April and May 2019. The trial involved more than 55,000 employers posting more than 200,000 job ads, eliciting over 5.5 million job applications. In this 2-arm trial, we tested the impact of the introduction of prompted choice into the job listing process compared to business-as-usual where there is no such prompt.

The job engine randomly allocated employers to view either the business-as-usual job listing template or a new template with an additional web page, prompting them to select the types of flexible working potentially available for the role. The flexible working options they selected were then displayed on the job advert for jobseekers to see .

The primary outcome measure was whether the resultant job posting offered flexible working options.

• to measure the flexible working options offered on job adverts, we used a web scraping algorithm that identified a list of predefined terms indicating flexibility (see Annex 1). The list was pre-tested to confirm that the terms were only used to signal the availability of flexibility^{[footnote 16]}
• to estimate the treatment effect, we used an ordinary least squares (OLS) regression model, clustered at the employer level and controlling for job function. No other covariates relating to the employers or the job adverts were available to us.

The secondary outcome measure was the number of applications per job advert, that is whether positions that offered flexible working arrangements attracted more applications.

• to capture this, we used data on the number of applications per listing within 2 weeks of the job posting, a timeframe suggested by the job site to capture the majority of applications
• we could not look at the gender differences in applications, because the job site does not collect data on applicant gender
• because the treatment was randomised at the employer level, applicants may have seen adverts from both the treatment and control groups. That may mean we overestimate the impact of offering flexible working options as the treatment may have ‘stolen’ applicants from the control group, which would not happen if the intervention was rolled out to the entire platform

2.3 Results

Job adverts published by employers who were exposed to the prompted choice treatment were 20% (7 percentage points, p<0.001) more likely to offer their positions as flexible, compared to the control group without such a prompt (Figure 1). This was a sizeable increase on a baseline of 34.5%.

Figure 1: Share of job adverts offering flexible working options (Primary outcome measure)

Share of adverts offering flexible working options	Control group	Treatment group
All flexible working job adverts	34.5%	41.5%

We ran exploratory analysis to measure the impact of the prompted choice on the offer of different kinds of flexible working arrangements. Exploratory analysis showed that the strongest effect was on an increased offer of flexitime (8.6 percentage points, p<0.001)^{[footnote 17]} , but all types of flexible working were affected, including part time (2 percentage points, p<0.001) (Figure 2).

Figure 2: Shares of adverts offering different types of flexible working (Exploratory analysis)

Share of adverts offering different types of flexible working	Control group	Treatment group
Any flexible working	34.5%	41.5%
Part time	32.5%	34.6%
Flexitime	4.3%	12.9%
Work from home	0.8%	2.6%

Looking at jobseeker behaviour, using the increase in flexible working advertising resulting from our intervention,^{[footnote 18]} we found that flexible jobs attracted on average 30% (p<0.05) more applicants. Job adverts without flexible working options attracted 23 applicants on average, so we estimate that they would have received 30 had they included flexible working options (Figure 3). The magnitude of this effect is much larger than we expected, and may be biased by one of the methodological limitations of the trial, which we discuss in the next section.

Figure 3: Impact of offering flexible working in job adverts on number of applicants

Type of job	Number of applicants per job
No flexible working	23
Flexible working	30

2.4 Limitations

There are 4 limitations that may bias the results presented in this report or the interpretation of their impact:

• there was a sample imbalance between the trial arms;
• the effect of offering flexible working on the number of applications may be overestimated;
• the effect of the additional message inserted into the treatment by the job site cannot be distinguished from the effect of the prompted choice; and,
• we cannot say whether the intervention resulted in more candidates actually being offered roles on a flexible basis

Firstly, despite having correctly randomised treatment assignment to employers, we found strong evidence of an imbalance between our treatment and control group on job function (see Annex 2), an internal classification the job site uses to reflect the tasks an employee is expected to be doing in the position (for example drivers, medical nurses or human resources). It is likely that this imbalance was driven by the fact that it was employers who were allocated into the different trial arms, rather than job adverts, coupled with the fact that advertisers tend to post a highly variable numbers of adverts. For this reason, we control for job function in all of our regressions, which accounts for the influence of job functions on our outcome measures, meaning that this imbalance should not directly bias our results.

However, this also means that we cannot be fully confident that the trial was balanced on other unobservable characteristics. For instance, it is possible that adverts in the treatment group are for jobs with a workplace culture that is more open to flexible working, which would bias our estimate upwards (so the effect we are reporting will be higher than the true effect). To check the possible impact of any further imbalance, we looked at whether excluding job function from the primary and secondary analysis regression makes a material difference to the results and found that it does not have such an effect. In other words, this suggests that treatment and control groups may not be different in a way that should matter substantively for our results. This is likely to be because the general size of any differences in job functions between treatment and control groups is rather small, even though it is statistically significant.

The second limitation relates to the constraints of our trial design in estimating the impact of offering flexible working options on the number of applications, our secondary outcome measure. The most reliable causal estimate of this relationship would ideally involve randomising jobseekers on the job site to observe different job adverts (some with flexible working options, and some without). Because the treatment was randomised at the employer level, and because it was deemed unethical to show different information about the same job to its applicants, the same jobseeker could have seen adverts in both the treatment and control groups. As such, if applicants in our trial chose between job adverts, it is possible that treatment group adverts “stole” applicants from control group adverts. Our estimate may therefore overstate the true impact of our intervention because this ‘stealing’ would not occur if the intervention was rolled out to the entire platform (as applicants then only see ‘treated’ job adverts). We may further investigate the magnitude of this ‘stealing’ in the final report.

Thirdly, the job site’s designers inserted an additional sentence into the prompted choice page on the job listing template that may have influenced advertisers. It read ‘All the fields below are optional, but including this information may strengthen your job post.’ The statement implied that mentioning flexibility could have a positive impact on the effectiveness of the listing. However, at the same time, it highlighted that it was optional, that is entirely up to the employer to decide whether to mention flexibility. First, this means that we cannot distinguish between the impact of the promoted choice, and this messaging. But more specifically, it means that we cannot say in which direction the messaging may have influenced our results. On one hand, it could increase an employer’s willingness to mention flexibility in the expectation that it will increase the attractiveness of the advert. On the other hand, highlighting the optional nature of the choice could discourage employers from making such a commitment.

Lastly, it should be noted that the aim of our intervention was only to encourage employers to advertise jobs as flexible and not to offer flexibility at the point of hiring. So while we can say that the intervention had a clear causal impact on offering flexibility on job adverts, we cannot tell whether this translated into an increase in the actual offer of flexibility at the point of hire because we did not measure this. However, we think it is likely that the intervention made it easier for applicants to start a conversation about the availability of flexibility, and that this translated in some cases into more people being hired on a flexible basis.

2.5 Implications

Subject to the limitations described above, this trial shows promising results whereby small changes to the choice architecture of job postings can encourage employers to advertise more jobs with flexible working options. Furthermore, jobs advertised in this way tend to attract more jobseekers.

Below, we hypothesise about the exact causal mechanism behind the success of our intervention. Looking at employer behaviour, a plausible explanation is that 4 key drivers were at play:

• recall: employers may have listed existing flexible working options that they already offer because they recalled what they can provide when reminded by the prompt
• improved transparency: employers willing to provide flexibility were compelled to offer this transparently
• increased supply: some employers may have been prompted to offer flexible working because they thought it would benefit them, when they would not have considered it previously. Also, the related statement inserted by the job site’s designers about the potential to strengthen the job advert may have contributed to this
• ease: being able to simply click on a pre-filled list of flexible working options in order to display them in the job advert may have increased the rate of employers doing so

As for applicant response, we think more jobseekers applied to flexible jobs because they prefer flexibility, appreciate employer openness on the availability of flexible working options and perhaps because they take these as a proxy for job and employer quality. Given that women are more averse to uncertainty and twice as likely as men to work flexibly due to the gendered division of labour, we can speculate that they may have been more likely to apply for the jobs advertised with flexible working options. However, we could not verify this hypothesis due to the lack of data on gender.

Our clear positive result is a new step in understanding how to increase the supply of flexible jobs in the UK, and possibly beyond. Making flexible working more widely available and offered from day one of a new job has the potential to help normalise flexible working for both women and men. By reducing the barriers for job applicants in asking for and justifying their need for flexible working arrangements, we may see a decoupling of flexible working arrangements as a working pattern which is mostly granted to or demanded by mothers. In turn, such working patterns may enable both women and men to thrive in roles that can better accommodate their wellbeing and their caring duties.

3. Round 2

3.1 Background

Following the Round 1 trial, we decided to run a second field experiment with Indeed to confirm our findings and try to boost the effectiveness of the intervention. Firstly, remarkably strong results in Round 1 and their inevitable implications for policy-making warranted a second trial to replicate our findings, important given the context of pervasive replication crisis in social research. Secondly, whilst we have seen sizeable effects of the prompt on both mentions of flexibility in jobs ads and applications, research suggested other potential behaviourally-informed approaches that might feasibly strengthen our impact. It should be noted that running 2 Rounds for replication and further research purposes was intended from the project onset.

3.2 Intervention design and test methodology

In continued partnership with Indeed and Harvard Business School, we conducted a second large field randomised controlled trial between September and December This time, the trial included almost 100,000 employers posting over half a million job ads, prompting over 14 million job applications. We designed a 4-arm trial, where we re-tested the impact of adding the prompted choice in the job listing template (Treatment 1) and also the impact of additional behaviourally-informed messaging highlighting flexible working as a legal right (Treatment 2) and gender inclusivity, as both men and women prefer flexibility (Treatment 3). As in Round 1 these were compared to business-as-usual where there is no such prompt (Table 1). The additional messaging was added on the top of the ‘prompted choice’ page in the job advert template. The specific messages were selected by GEO, from a shortlist provided by BIT in collaboration with Harvard Business School.

Table 1. Round 2 trial arms

Arm	Title	Detail
Control	Business- as-usual	No extra page on flexible working in the job ad template
Treatment 1	Prompted choice	“All the fields below are optional, but including them may strengthen your job post. What flexible working options would you consider for this role?” ~ identical to Round 1
Treatment 2	Legal right	“UK employees have the right to demand flexible working, subject to eligibility.” Rationale: A previous BIT trial on shared parental leave found that highlighting legal rights can increase uptake.[footnote 19]
Treatment 3	Gender inclusivity	“84% of male and 91% of female full-time workers already work flexibly, or want to. Advertising flexible working options in your job listing could help improve inclusion in your organisation.” Rationale: Correcting misperceptions about other people’s views and behaviours can help normalise them.[footnote 20] Targeting the myth that flexible work is for women only may prove effective for tackling gender bias.

Other treatments considered included highlighting employer social norms and providing feedback from Round 1.

The outcome measures and analytical strategy remained the same as in Round 1.

For a reminder:

• the primary outcome measure was whether the job posting resulting from the job listing process offered flexible working options. This was measured using a web scraping algorithm that identified a list of predefined terms indicating flexibility, to make sure we are correctly estimating the prevalence of flexibility in our control group. To estimate the treatment effect, we used an OLS regression model, clustered at the employer level and controlling for job function, with no other covariates available
• the secondary outcome measure was the number of applications per job advert that is, whether positions that offered flexible working arrangements attracted more applications, within 2 weeks of the job posting. Due to randomisation at the employer level, applicants may have seen adverts from both the treatment and control groups which could lead to overestimation of the size of the effect of flexibility on applications (Limitations in Round 1). For a fuller account of outcome measures and analysis strategy, please refer to Round 1

3.3 Implementation challenges

Round 2 faced several implementation challenges, one of which resulted in the evaluation no longer being considered a randomised controlled trial. Each challenge is detailed below with comments on how the challenge affected our analyses and the interpretation of the evidence.

The primary challenge was that the control group was not one of the randomisation arms due to an implementation error. To address this Indeed provided additional data on unexposed advertisers from the same time period, as a comparison group. When we conducted balance checks, we found imbalances in employers and job types between the treatment arms and this comparison group. This means that we cannot be certain that any differences between the treatment and the control groups are attributable to the intervention alone, as they could be potentially driven by other systematic differences between the participants present at the baseline. Therefore, the strength of the evidence generated in this part of the evaluation should be considered, equivalent to a weakly matched quasi-experimental design (QED) study. That is, we cannot make strong inferences about the causal impact of our treatment because it was not assigned completely at random. In the following section, we present a rapid summary of the general direction of the results but caution that these should be considered against the background of the implementation challenges faced.

Shorter trial length reduced sample size.

The trial had to be terminated earlier than originally intended, due to an unanticipated update of Indeed’s job advert template. The new version of the template was rolled out, without the treatment, reducing the sample of prospective participants during an already slower winter period. As a result, we received data from around 64,000 adverts per arm, compared to the planned circa 84,000 adverts per trial arm. At the same time, we also received more data from the control group than requested (303,796 instead of 84,000) (see below).

Incorrect implementation of trial arms.

One of the trial arms — ‘prompted choice’ — was implemented twice in error, instead of including the control group, with a business-as-usual job advert template. For the main analysis, we pooled the identical treatment arms and analysed how the 2 compared as part of the exploratory analysis.

Inclusion of ‘old’ employers (already treated in Round 1).

We originally intended to test Round 2 on a sample of ‘new’ employers, who were not previously exposed to the treatment within Round 2. This was to avoid exposing employers to more than one treatment, so that we could isolate the treatment effects from each other. However, ‘old’ employers were included alongside ‘new’ employers in error. In particular, 92.6% of ads were from advertisers who participated in Round 1 of the experiment. In the analysis, we looked at both merged and separate estimates for the 2 groups.

“Not offered” ads risked confounding the flexibility estimate.

When employers selected “Not offered” from the list of flexible working options in the prompted choice box, this was automatically displayed on their job ad as follows: “Flexible working options available: Not offered.” This posed a challenge to the original web scraping strategy, because the string of words “flexible working” would be picked up and could be inflating our estimate of the offer of flexibility and hence, overestimation of the treatment effect. At the same time, “Not offered” could be selected alongside other flexible working options, so we did not want to discount it altogether. We were able to correct for this issue by changing the web scraping logic, so that the job ad had to contain other mentions of flexibility beyond this string to be counted as flexible.

3.4 Indicative results

Subject to these important caveats, we found results consistent with Round 1 findings, albeit with small differences in effect sizes. However, we didn’t find any evidence that additional behaviourally-informed messaging would be more effective than a prompted choice page alone. Below we outline the summary of findings, for full results, consult Appendix 2.

Employers exposed to the prompted choice for the first time appear to be twice as likely to advertise jobs as flexible compared to Round 1 (an increase of 11 percentage points). Within the unplanned exploratory analysis involved with correcting implementation errors, we looked at any differences between ‘old’ and ‘new’ employers. We also found that whilst ‘old’ employers previously exposed to the intervention in Round 1 were only slightly less likely to advertise jobs flexibly than the first time (an increase of 5 percentage points). Overall, the full sample (old and new) which received treatment in Round 2 were 6 percentage points more likely to advertise jobs flexibly and such offers received more applications (19 percent).^{[footnote 21]} The additional alternative messaging on the prompted choice page was not more effective than the baseline effect of the prompt.

Whilst these results were broadly consistent with Round 1, their integrity might have been compromised by the aforementioned implementation challenges. Firstly, our balance checks flagged some imbalances between the control and the treatments, in terms of both job type and employers. Although we control in our analysis for the type of the posted job, it is still possible that this procedure does not sufficiently account for the differences in types of employers between control and treatment group. Secondly, when we compared the effect of the treatment arms against one another on the primary outcome, we found a difference between 2 presumably identical treatment arms (1 percentage point). We have not been able to confirm any plausible explanation for this, but this finding provides further confirmation that the results should be considered with caution. For full results, consult Annex 2.

3.5 Implications

Due to severe implementation challenges, randomisation of the trial was compromised which means our findings do not reach the highest robustness standards.

Nevertheless, the findings of this study do not contradict the Round 1 results but add additional, if weaker, support. The general direction of these findings was positive with broadly the same magnitudes of effect from Round 1. We find no reason to change the conclusion from the Round 1 trial that prompting employers to select flexible working options during job advert placement on jobsites can increase the number of flexible jobs being offered. Such jobs are also more likely to attract more applicants than roles without any offer of flexibility. Whilst we cannot speak to the gender split of applicants on the basis of this trial, we explored this in the follow-up online experiment (Section 3 Online trial).

More should be done, to ensure employers offer flexibility and that it is signalled transparently so that women and men can access flexibility to improve their work-life balance, care burden and wellbeing. Such normalisation of flexibility over time should help to tackle the part-time penalty, one of the key drivers of the gender pay gap.^{[footnote 22]}

4. Online Trial

4.1 Background

We ran a follow-up online experiment to compensate for some of the limitations of the field trials with Indeed. Firstly, the field trials were randomised at the advertiser level rather than at the candidate level. This meant that whilst we could say that job adverts that offer flexibility attract more candidates, we couldn’t isolate the impact that offering flexibility has on individual candidates. This was likely to contribute to the overestimation of the effect of flexibility on applications, as the same candidate could have been exposed to job ads from both the control and the treatment. Secondly, we were not able to access information about the gender of applicants, because Indeed does not collect this data nor was it comfortable with inferring applicants’ gender from their names. This meant that we could not say whether the offer of flexibility elicits different reactions from men and women.

4.2 Intervention design and test methodology

Working with BIT’s online experimental platform Predictiv, we conducted an online randomised controlled trial in November 2019. The final trial included just over 5,000 participants who went through a job shortlisting exercise, followed by additional questions on the perceived organisational fit with the fictional employer advertising the job, and perception of whether the fictional employer offered reduced hours and choice of hours based on the job advert. Finally, we asked participants a range of demographic questions including the industry they worked in, their education, whether they had child caring responsibilities, and their current employment status.

Participants from Predictiv’s online panel are broadly representative of the UK’s working population age (50% under 40) and income (50% with a household income of less than £30,000). Participants had to either be economically active (have a job and have searched for a job in the last year or will search for a job in the next year) or if inactive, likely to look for a job in the next year.

They were asked to imagine that they are looking for a new job and need to shortlist the most interesting offers. They were then randomly allocated to see one of 5 versions of a job advert: a control ‘business-as-usual’ job advert (generically similar to many job ads today that do not mention flexible working), and 4 different versions with varying degrees of flexible working offerings and specificity of the offers.

Table 2. Online trial intervention arms

Condition	Description	Rationale
1.	Control (Full-time) Schedule: Full-time.	Mirroring the majority of job adverts available today.
2.	FW vague Schedule: Full-time. Happy to talk flexible working	Reflecting BEIS’s “Happy to talk flexible working campaign”, which encourages employers to add this sentence to their job ads.
3.	FW specific Schedule: Full-time; Flexible working options: - working from home - compressed hours - flexitime	This arm specifies which options for flexible working will be available on the job. It also mirrors a possible outcome for an employer that responded to the intervention we ran with Indeed.
4.	FW specific + part-time Schedule: Full-time; part-time. Flexible working options: - working from home - compressed hours - flexitime	Offering part-time work in addition to specific flexible working offering
5.	FW specific + part-time + inclusive sentence Schedule: Full-time; part-time. Flexible working options: - working from home - compressed hours - flexitime This job is offered on a part-time basis because we want the best people for our roles, and we recognise that sometimes those people aren’t available full time.	Adding an ‘inclusive sentence’ akin to diversity statements, to combat the stigma associated with part-time work.

The primary outcome measure was the willingness to shortlist the job (Answer to the question: How likely are you to shortlist this job?), on a scale of 0 (not at all likely) to 5 (extremely likely).

Given women are more likely to have caring responsibilities and hence more likely to demand flexibility, we expected to see a stronger effect of offerings of flexibility on women. Moreover, we expected women to also prefer the more specific statements, given their aversion to ambiguity.^{[footnote 23]}

The secondary outcome measures, using the same 0 to 5 scale, included:

• organisational fit (How likely are you to experience a good fit with people in this organisation?). As mentions of flexible working signal more inclusivity to women, we expected this to boost women’s expectations of a good organisational fit and potentially reduce it for men, given flexibility is often associated with women.
• reduced hours (How likely do you think this employer is to allow you to reduce your working hours?)
• freedom to choose hours (How likely is this employer to allow you to choose your working hours?)

4.3 Results

For women, all treatments statistically significantly increased the self-reported likelihood to shortlist the job compared to the control (no mention of flexible working). In addition, the specific description statistically significantly increased the likelihood to shortlist the job compared to the vague description. We observed the same pattern for men, except that the likelihood to shortlist the job did not significantly differ when a vague description of flexible working was included compared to no description. Including part-time in the working schedule directionally decreased the likelihood of men to shortlist the job compared to the specific description alone but this result was not statistically significant.

Figures 4: The effect of flexible working on likelihood to shortlist the job

Types of flexible working descriptions	Likelihood of women shortlisting the job	Likelihood of men shortlisting the job
Control	3.6%	3.7%
Vague description	3.8%	3.8%
Specific description	4.1%	4.2%
Specific plus part-time	4.2%	4.0%
Specific, part-time and inclusive sentence	4.1%	4.0%

As for demographic differences, being primary carer, unemployed or on benefits increases the likelihood of shortlisting the job offering flexibility, for both men and women. We found primary adult carers were more likely to shortlist the job by 0.2 units (on the 5-point scale), and primary child carers were 0.13 units more likely to shortlist the job (both significant at the 1% level). However, it seems that after the age of 40, interest in flexibility declines across both sexes. As for other factors, education and location seems to influence men. Those with only primary education and from the South and East England (in comparison to London) are less likely to shortlist jobs that offer flexibility. We found that all education levels above primary were more likely to shortlist flexible jobs by 0.7 units, whilst South East and East England were less likely to shortlist them by 0.2 units. Income or ethnicity did not seem to influence the likelihood of shortlisting flexible jobs in our experiment.

Looking at the secondary outcomes, we found that mentions of flexibility boosted the perceived organisational fit for both men and women, though even more so for women than men. The perception that employers will allow reduced hours and that the employee will be able to choose their hours, was also higher when flexibility was mentioned, for both women and men. The order of magnitude and strength of the effect were similar in both.

For women, the specific description and inclusion of ‘part-time’ on the job advert was the most effective across these measures of organisational fit, reduced hours and choice of hours. For men, the specific description of flexibility with part-time and an inclusive sentence performed best for organisational fit and directionally best for reducing hours. All specific descriptions were effective in increasing the perceived likelihood that the employer will allow a choice of hours (see Annex 3 for all figures).

Figures 5: The effect of treatments on the likelihood of experiencing organisational fit for women and men

Types of flexible working descriptions	Likelihood of women experiencing organisational fit	Likelihood of men experiencing organisational fit
Control	4.0%	4.0%
Vague description	4.2%	4.1%
Specific description	4.3%	4.2%
Specific plus part-time	4.3%	4.2%
Specific, part-time and inclusive sentence	4.3%	4.3%

4.4 Limitations

Various academic studies show that results from designs that simulate the decision environment people face in practice map on closely to behaviour outside of the experiment. This includes work on voting^{[footnote 24]}, credit card repayment decisions, and anti-social behaviour such as fare-dodging in public transport^{[footnote 25]} and accepting bribes^{[footnote 26]}. However, decision-making in a simulated environment is also subject to some important limitations. These experiments often feature survey-style questions that are likely to be subject to participants responding in a way to be viewed more favourably (so-called social desirability bias). Moreover, participants state what they would do, in reaction to materials, but intentions do not always translate into actions (intention-action gaps). This suggests that most results from online experiments should be understood rather as signals of the direction of the effect and of the relative performance of different interventions. Given intention-action gaps and the difference between online and real world settings, we should treat the size of the effect as being on the upper bound of effects we are likely to see in the real world (unless the parallels between the actual and online behaviour are stronger).

However, we expect our findings of the relative differences between the variations of the job description to hold in the real world.

4.5 Implications

Notwithstanding the limitations of online testing, this additional trial shines further light on the results of the field trials with Indeed, showing that it is probable that both men and women were equally likely to apply more for jobs mentioning flexibility.

This finding is consistent with existing surveys showing that men and women are equally interested in flexibility and helps to dispel some of the typical stereotype that flexibility is just for working mothers.

Moreover, this trial confirms that more specificity in the description of flexible working elicits more interest from jobseekers. This is likely because providing a clear commitment to specific forms of flexibility by employers makes such an offer seem more reliable and trustworthy.

5. Annexes

5.1 Annex 1: List of web scraping terms

Included terms:

Flexitime

Flexible start and finish time

Flexible start time

Flexible finish time

Flexible approach to working hours

Flexible hours

Staggered hours

Compressed hours

Annualised hours

Job share

Job sharing

Part-time

Part time

Phased retirement

Working from home

Work from home

Remote work

Remote working

Work remotely

Home work

Home working

Flexible work (except if “Flexible work style”) Flexible working (except if “Flexible working style”)

Flexible working arrangement

Flexible working options

Flexible working hours

Flexible schedule

Flexible scheduling

Work hours flexibly

Flexible ad hoc hours

Flexible on days and hours

Flexible days and hours

Flexible working approach

Ability to split hours

Terms included if ‘full-time’/‘full time’/‘part-time’/‘part time’ were not found:

1 day per week

2 days per week

3 days per week

4 days per week - All possible permutations of the above phrases using a) any numeric value <=4 instead of ‘1’/’2’ etc.; b) ‘a’ instead of ‘per’; c) ‘every’ instead of ‘per’; d) ‘each’ instead of ‘per’; e) ‘weekly’ instead of ‘per week’.

1 hour per day

2 hours per day

3 hours per day

4 hours per day

All possible permutations of the above phrases using a) any numeric value <=4 instead of ‘1’/’2’ etc.; b) ‘a’ instead of ‘per’; c) ‘every’ instead of ‘per’; d) ‘each’ instead of ‘per’; e) daily instead of ‘per day’; f) ‘hr’/’hrs’ instead of ‘hour’/’hours’; g) ‘h’ instead of ‘hours’.

Finally, the same permutations as for days per week, but for anything less than or equal to 30 hours per week.

Terms excluded (even when matches were found for the above):^{[footnote 27]}

Zero hours

Zero hour

0 hours (excluding cases where another digit appears immediately before 0, i.e. 10/20 etc.)

0 hour (excluding cases where another digit appears immediately before 0, i.e. 10/20 etc.)

Flexible Working Options Available - Not offered

5.2 Annex 2: Round 2 results

Results of Round 2 were very similar to Round 1, with small differences in effect sizes, but we didn’t find any evidence that additional behaviourally-informed messaging would be more effective than a prompted choice page alone.

Due to an implementation error, many of ‘old’ employers already exposed during Round 1 were included again in Round 2 and hence doubly exposed to the intervention. Specifically, 74.2% of advertisers in the treatment groups participated in Round 1. Furthermore, advertisers who participated in Round 1, posted 92.6% of all ads in the trial, suggesting an imbalance in the average number of job postings between those who participated in Round 1 and those who did not. For clarity, we present first the effect on ‘new’, never previously exposed employers, as this is the cleanest comparison to Round 1, originally intended in the experiment design. This is followed by exploratory analysis on the treatment effect among ‘old’ employers, double exposure due to their participation in Round 1, and lastly, the merged estimate for the 2 groups, as it happened.

Generally, we find that the results are similar to those found in the Round 1 trial; however, given the implementation challenges, they should be considered with caution.

We found that job adverts of ‘new’ - first time exposed - employers were 29% (11 percentage points) more likely to offer their positions as flexible, compared to the control group with a baseline of 36% (Figure 6). In Round 1 we saw a 20% increase (7 percentage points), on a slightly lower baseline of 34.5% in the control group. Looking at ‘old’ employers who were exposed to the treatment in both rounds, we found that an effect of broadly the same magnitude (on average 5 percentage points) remained (Figure 7). This was similar in size to that seen in Round 1 though slightly weaker. This may be because the novelty of the prompt was not there in the second round for these employers.

Finally, if we merge ‘new’ and ‘old’ employers, we get a global effect for Round 2 of a 17% (6 percentage points) increase of employers becoming more likely to offer their positions as flexible, compared to the control group with a baseline of 36% (Figure 8).

Note: Significance marks (*) in the following tables show that outcomes differ compared to the comparison group but not that these differences are causally attributable to the treatment.

Figure 6: Treatment effect among ‘new’ (first exposed) advertisers (did not participate in round 1)

Treatment type	Probability
Control	36.4%
Prompted choice (PC)	46.8%*
PC and legal right	47.3%*
PC and gender equality	46.8%*

N = 19133 (excluding control group) *p<0.01 exploratory analysis

Figure 7: Treatment effect among ‘old’ (double exposed) advertisers (participated in round 1)

Treatment type	Probability
Control	36.4%
Prompted choice (PC)	41.2%*
PC and legal right	40.5%*
PC and gender equality	41.1%*

N = 238275 (excluding control group) *p<0.01 exploratory analysis

Figure 8: Treatment effect among both ‘old’ and ‘new’ advertisers merged together

Treatment type	Probability
Control	36.4%
Prompted choice (PC)	42.4%*
PC and legal right	42.0%*
PC and gender equality	42.4%*

N = 561204 (excluding control group) *p<0.01 exploratory analysis

Looking at the effect on different kinds of flexible working arrangements, we found impact across all types of flexibility. Consistent with Round 1 findings, our exploratory analysis showed that the biggest effect was on an increased offer of flexitime (10 percentage points), with smaller increases of mentions of part time or working from home (1 percentage points) (Figure 9).

Figure 9: Shares of adverts offering different types of flexible working (Exploratory analysis)

Type of flexible working	Treatment probability of advertising	Control probability of advertising
Part-time	35.3%*	33.8%
Flexitime	9.6%*	0.1%
Work from home	2.1%*	0.5%

N = 561204 (excluding control group) p<0.01 exploratory analysis

As for jobseeker behaviour, we found that jobs mentioning flexibility attracted on average 20% more applicants. Whilst jobs without flexible working options received 22 applications on average, we estimate that flexible jobs could attract up to 26 applications (Figure 19). This is broadly in line with our Round 1 findings, albeit with considerably smaller effect size (in Round 1 we found a 30% (p<0.05) increase, moving from 23 to 30 applicants on average). Seasonality could partly help to explain this difference, if fewer candidates were looking for flexibility during the winter period when the job market was less active. Indeed has not seen any evidence for seasonality of searchers for flexibility. However, we cannot rule this out. Given the paucity of data on job adverts and employers, we could not investigate what, if any, underlying factors, might be driving this difference.

Again, it should be noted that the still sizeable effect is a subject to overestimation, because the same applicant could in theory see both results from the control and the treatment, leading to ‘stealing’ of applicants to treatment (see Limitations in Round 1).

Figure 10: Impact of offering flexible working in job adverts on number of applicants

Type of flexible working	Number of applications
Flexible working	26.2*
No flexible working	22.1

N = 561204 (excludiing control group) *p<0.01 exploratory analysis

We also explored the treatment effect of the 2 unmerged prompted choice arms. To remind, the trial included 2 identical prompted choice arms instead of one, due to an implementation mistake, as outlined in ‘Implementation challenges’. Whilst we used a merged estimate for our main results, we also looked at the effect of the treatment arms against one another on our primary outcome, finding a difference was between the 2 identical treatment arms (1 percentage point). We have not been able to confirm any plausible explanation for this, but this finding provides further confirmation that the results should be considered with caution.

Figure 11: Comparison of the 2 prompted choice arms

Treatment type	Probability
Control	36.4%
Prompted choice 1 (PC)	41.9%*
Prompted choice 2 (PC)	42.9%*
PC and legal right	42.0%*
PC and gender equality	42.4%*

N = 561204 (excluding control group) *p<0.01 exploratory analysis

5.3 Annex 3: Online trial - all outcome tables

Figures 12: Effect of treatments on perceived likelihood of employer allowing you to reduce your working hours for men and women

Types of flexible working descriptions	Likelihood of employer allowing reduced hours
	Women	Men
Control	3.3%	3.5%
Vague description	3.9%	3.8%
Specific description	4.0%	4.1%
Specific plus part-time	4.3%	4.0%
Specific, part-time and inclusive sentence	4.3%	4.2%

Figures 13: Effect of treatments on perceived likelihood of employer allowing you to choose your working hours for women and men

Types of flexible working descriptions	Likelihood of employer allowing you to choose your hours
	Women	Men
Control	3.2%	3.4%
Vague description	3.9%	4.5%
Specific description	4.3%	4.8%
Specific plus part-time	4.4%	4.8%
Specific, part-time and inclusive sentence	4.3%	4.8%

6. Acknowledgements

Authors: Kristina Londakova, Vivek Roy-Chowdury, Filip Gesiarz, Hannah Burd, Rony Hacohen, Abigail Mottershaw, Janna Ter Meer and Tiina Likki.

Professor Iris Bohnet (Harvard Kennedy School), Associate Professor Mike Luca (Harvard Business School), Dr Heidi Liu (Harvard Kennedy School) provided expert advice on the intervention design.

Elisabeth Costa and Dr Kimberly Bohling (BIT) provided expert advice on the intervention and trial design and assisted with the quality assurance.

Laura Hegarty, Pawel Adrjan, David Freddolino, Emily Gipple, Milos Curcic and Alassane Ndour from Indeed supported the implementation of both field trials.

1. By flexible working we mean all types of flexibility – including the amount of hours worked (for example part-time) the working hours (such as flextime; compressed hours), the location of the work (for example working from home, working remotely) and other arrangements (such as job sharing). ↩

2. Both currently unemployed (people without a job who have been actively seeking work within the last 4 weeks and are available to start work within the next 2 weeks) and economically inactive (people not in employment and not been seeking work within the last 4 weeks and/or they are unable to start work in the next 2 weeks). ↩

3. Defined as permanent positions that pay £20,000 or more per year. ↩

4. Borghans, L., Heckman, J. J., Golsteyn, B. H., & Meijers, H. (2009). Gender differences in risk aversion and ambiguity aversion. Journal of the European Economic Association, 7 (2-3), 649-658. ↩

5. The employee chooses when to start and end work (within agreed limits) but works certain ‘core hours’. ↩

6. Borghans, L., Heckman, J. J., Golsteyn, B. H., & Meijers, H. (2009). Gender differences in risk aversion and ambiguity aversion. Journal of the European Economic Association, 7(2-3), 649-658. ↩

7. Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press, New Haven, CT. ↩

8. Kahneman, D. (2011). Thinking, fast and slow. Macmillan. ↩

9. BIT’s own analysis of the job website data. ↩

10. Samuelson, W., & Zeckhauser, R. (1988). Status quo bias in decision making. Journal of risk and uncertainty, 1(1), 7-59. ↩

11. Croson, R., & Gneezy, U. (2009). Gender differences in preferences. Journal of Economic literature, 47(2), 448-74. ↩

12. Carroll, G. D., Choi, J. J., Laibson, D., Madrian, B. C., & Metrick, A. (2009). Optimal defaults and active decisions. The quarterly journal of economics, 124(4), 1639-1674. ↩

13. Gee, L. K. (2018). The more you know: information effects on job application rates in a large field experiment. Management Science. ↩

14. Leibbrandt, A., & List, J. A. (2014). Do women avoid salary negotiations? Evidence from a large-scale natural field experiment. Management Science, 61(9), 2016- 2024 ↩

15. He, H., Neumark, D., & Weng, Q. (2019). Do workers value flexible jobs? A field experiment on compensating differentials (No. w25423). National Bureau of Economic Research. ↩

16. For instance, we excluded terms such as ‘Flexible working style’, ‘Flexible work style’ or ‘Flexible approach’ used to ask for staff to be flexible as in ready to adapt to circumstances; or to require for them to be available flexibly, on short notice or working through weekends. Our pre-tests also showed that advertisers only mention flexibility when they do offer it (that is, not to list what they do not offer). We also excluded cases where employers directly indicated in the treatment group that flexible working was not offered using the following string: ‘Flexible Working Options Available: Not offered’. ↩

17. This is greater than the overall increase in flexible working advertising (7pp) because most of the increase in flexitime occurred among part-time jobs. ↩

18. We used an instrumental variable approach. ↩

19. Hacohen, R., Likki, T., Londakova, K. and J.Rossiter (2018). Return to work: Parental decision making. Government Equalities Office. ↩

20. Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007). The constructive, destructive, and reconstructive power of social norms. Psychological science, 18 (5), 429 - 434. ↩

21. Due to the significant implementation issues, we are only reporting observed effects and not p-values. ↩

22. Goldin, C. (2015). How to achieve gender equality. Milken Institute Review Q, 3 , 24-33. ↩

23. Croson, R., & Gneezy, U. (2009). Gender differences in preferences. Journal of Economic literature, 47(2), 448-74. ↩

24. Hainmueller, J., Hangartner, D., & Yamamoto, T. (2015). Validating vignette and conjoint survey experiments against real-world behaviour. Proceedings of the National Academy of Sciences, 112 (8), 2395 - 2400 ↩

25. Dai, Z., Galeotti, F., & Villeval, M. C. (2017). Cheating in the lab predicts fraud in the field: An experiment in public transportation. Management Science, 64 (3), 1081-1100. ↩

26. Gneezy, U., Saccardo, S., & van Veldhuizen, R. (Forthcoming). Bribery: Behavioural drivers of distorted decisions. Journal of the European Economic Association. ↩

27. Zero hour jobs are excluded because the flexibility of this job pattern is typically and primarily retained by the employer, rather than offering true flexibility to the employee ↩