Research and analysis

Road-testing remedies, experimental techniques and evaluation

Published 3 August 2017

The content is divided into ‘core reading’ and ‘further reading’ lists. Core reading captures fundamental and/or important documents of use primarily to inexperienced users while further reading lists capture more specific material intended for experienced practitioners.

1. Overview of the key approaches

1.1 Core reading

Document	Topics covered	Notes
EC (2013), ‘Applying Behavioural Sciences to EU Policy-making’	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs.	Explains when and how behavioural science should be applied to policy making including an overview of the different research methods.
EC (2015), ‘Seven Points to Remember when Conducting Behavioural Studies in Support of EU Policy-making’	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs.	This policy brief is a follow-up to EC (2013) on applying behavioural sciences and aims to provide policy-makers with practical guidance for planning and managing a behavioural study.
Francesco Guala (2005), ‘The Methodology of Experimental Economics’	Experimental economics, remedy testing.	This book provides a comprehensive analysis and critical discussion of the methodology of experimental economics. It outlines the fundamental principles of experimental inference in order to investigate their power, scope and limitations.
Bardsley, Cubitt, Loomes, Moffatt, Starmer and Sugden (2010), ‘Rethinking the Rules’	Economic behaviour, experiments, RCTs	The book reviews a wide range of experiments, all carried out with the intention of contributing in some way or other to the understanding of economic behaviour.

1.2 Further reading – guidance

Document	Topics covered	Notes
OFT1099 (2009), ‘Road testing of consumer remedies’ (PDF, 798 KB)	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs, simulation, quasi-experiments, econometrics.	This report provides rich information on the different research methods to assess consumer remedies prior to their implementation. Covers lab and field trials under the broader category of economic experiments.
Commonwealth of Australia, Department of the Prime Minister and Cabinet (2016), ‘Guide to developing behavioural interventions for randomised controlled trials’ (PDF, 806KB)	Remedy testing, RCTs, pre-testing.	Covers 4 project stages to developing behaviour interventions for RCTs focusing on the first two stages: - Discovery - defining outcomes of interest; establishing whether they can be measured; and whether standardised interventions are feasible and can be delivered to a large enough randomised population. - Diagnosis – understanding behaviour and how interventions might influence this

2. Experimental and survey-based techniques

2.1 Core reading

Document	Topics covered	Notes
List J, Sadoff S and Wagner M (2012), ‘So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design’	RCTs, lab trials, field trials, experiments.	Develops several simple rules of thumb that researchers can apply to improve the efficiency of their experimental designs. This is a more technical article on experimental design than the guidance documents listed in the overview section above. Also includes empirical examples from the literature.
CC/OFT (2011), ‘Good practice in the design and presentation of consumer survey evidence in merger inquiries’	Surveys	While the focus of this guidance is on surveys in merger inquiries, the good practice principles it sets out have wider relevance to survey work for consumer remedies.

2.2 Further reading – selected academic papers

Document	Topics covered	Notes
Al-Ubaydli O and List J (2012), ‘On the generalizability of experimental results in economics’	RCTs, lab trials, field trials, external validity, generalisability.	This study provides an overview of experimental methods in economics, with a special focus on developing an economic theory of generalisability. It concludes that until we conduct more field experiments that build a bridge between the lab and the naturally-occurring settings of interest, we cannot begin to make strong conclusions empirically on the crucial question of generalisability from the lab to the field.
Mitchell G (2012), ‘Revisiting truth or triviality: The external validity of research in the psychological laboratory’. Perspectives on Psychological Science, 7(2), 109-117	RCTs, lab trials, field trials, external validity, generalisability.	Using 217 lab-field comparisons from 82 meta-analyses, this paper finds that the external validity of laboratory research differed considerably by psychological subfield, research topic, and effect size. Laboratory results from industrial-organisational psychology most reliably predicted field results, although results from IO labs still varied in their generalisability.

2.3 Further reading – case studies

Document	Topics covered	Notes
FCA (2016), ‘Full disclosure: a round-up of FCA experimental research into giving information’ (PDF, 1,239 KB)	Remedy testing, RCTs, online experiments, field trials.	This round-up paper presents the results of 5 field trials and 3 online experiments, which have tested the effect of interventions that draw on behavioural theory, such as increasing salience or personalisation. It also shares some practical lessons learnt.
Optimisa (2016), ‘Informing the development of communication tools designed to increase consideration of switching among PCA and SME customers’ (PDF, 3.1 MB)	Remedy testing, qualitative research.	Used as an input to the design of consumer remedies in the CMA Retail Banking Market Investigation (2016), including the recommendation to undertake RCTs on prompting customers to consider their banking arrangements.
CC (2009), ‘Payment Protection Insurance market investigation’, Appendices 3.9, 10.9, 10.10 and 10.11 and associated GfK NOP Survey on PPI Search Behaviour (PDF, 1,274 KB)	Remedy testing, simulation, surveys.	Includes using survey evidence as part of a simulation of the effects of remedies aimed at increasing the proportion of consumers that shop around for PPI.
FCA (2014), ‘Occasional Paper No. 3: How does selling insurance as an add-on affect consumer decisions?’	Remedy testing, online experiments.	This paper discusses the pros and cons of using behavioural experiments in creating regulatory policy. The paper illustrates this by describing an innovative experiment, which finds that when general insurance is sold as an add-on product to a primary purchase, this weakens consumers’ ability to discipline firms by shopping around and comparing products effectively. This in turn has implications for possible remedies.
FCA (2015), ‘High-Cost Short-Term Credit Price Comparison Websites: A behavioural study for the Financial Conduct Authority’ (PDF, 2.2 MB). Research by London Economics and YouGov	Remedy testing, online experiments, credit.	Used an online experiment to test the impact of imposing a series of potential standards on price comparison websites of high-cost short-term credit products.
Ofcom (2010), ‘Using experiments in consumer research’ (PDF, 913 KB)	Remedy testing, lab trials.	To assess the potential benefits and limitations of experiments, Ofcom commissioned this experimental study on devising effective ways of providing information on the price of telephone calls. Includes a discussion on using experiments in economics for regulatory policy.
OFT (2010), ‘The impact of price frames on consumer decision making (OFT1226)’ (PDF, 3.5 MB)	Remedy testing, lab trials.	This study examines the impact on consumer decision-making of different ways in which prices can be framed using a controlled experiment. It fed into the OFT’s study on the advertising of prices.
OFT (2013), ‘Partitioned Pricing Research – A behavioural experiment (OFT1501A)’ (PDF, 1,563 KB)	Remedy testing, lab trials.	Examines the impact of partitioned pricing on consumer decision-making. Forms part of a wider set of partition pricing research of the OFT in 2013. This research was related to the OFT investigations on airlines/payment surcharges.

3. Randomised controlled trials in the field

3.1 Core reading

Document	Topics covered	Notes
Behavioural Insights Team (2012), ‘Test, Learn Adapt’	Remedy testing, RCTs, field trials.	The paper argues that RCTs, should be used much more extensively in public policy to enable policymakers to test which interventions are most effective. It also sets out 9 separate steps that are required to set up any RCT.
List A (2012), ‘Why economists should conduct field experiments and 14 tips for pulling one off’	RCTs, lab trials, field trials, external validity, generalisability.	Offers an overview of experimental methods in economics from laboratory experiments to field experiments with a focus on field experiments.

3.2 Further reading – practical considerations

Document	Topics covered	Notes
J-PAL (2013), ‘Introduction to evaluations’	RCTs, evaluation.	Covers the what, why, who, when, and how of randomised evaluations.
IPA (2015), ‘Evaluating Financial Products and Services in the US: A Toolkit for Running Randomized Controlled Trials’ (PDF, 1,916 KB)	RCTs	Introductory toolkit aimed at researchers seeking to conduct RCTs with financial service providers. This toolkit also has relevance to other sectors.
Australian Government Department of Industry and Science (2015), ‘Randomised Controlled Trials and Industry Program Evaluations’	Remedy testing, RCTs, field trials, evaluation, generalisability.	Develops a framework that can be used to assess the suitability of evaluating policy interventions using RCTs.
Cartwright, Deaton (2016), ‘Understanding and Misunderstanding Randomised Controlled Trials’	RCTs, generalisability.	Raises a number of concerns relating to the ability of RCTs to provide robust and/or generalisable evidence.

3.3 Further reading – case studies

Document	Topics covered	Notes
FCA (2016), ‘Occasional Paper No. 19: Attention, Search and Switching: Evidence on Mandated Disclosure from the Savings Market’	Remedy testing, RCTs, field trials, switching triggers, retail banking, field trials.	This paper describes the findings of a series of field trials that explored how disclosure-type regulatory interventions could encourage consumers to switch to higher-paying savings accounts.
FCA (2015), ‘Occasional Paper No.12: Encouraging consumers to act at renewal: evidence from field trials in the home and motor insurance markets’	Remedy testing, RCTs, field trials, switching triggers, retail insurance.	This paper presents the results of field trials testing the potential for improved renewal notices to encourage consumers to switch or negotiate their home or motor insurance policy at renewal.
Bertrand M and Morse A (2011), ‘Information Disclosure, Cognitive Biases and Payday Borrowing’	Remedy testing, RCTs, field trials, credit.	Uses a field experiment to assess whether psychology-guided information disclosure induces borrowers to lower their use of high-cost debt.

4. Evaluation

4.1 Core reading

Document	Topics covered	Notes
HM Treasury (2011), ‘The Magenta Book’	Evaluation	Guidance on what to consider when designing an evaluation.

4.2 Further reading – guidance

Document	Topics covered	Notes
Ofcom (2013), ‘A review of consumer information remedies’	Evaluation, information remedies.	Section 6 of this review explores the issues around evaluating the effectiveness of an information remedy.
National Audit Office (2011), ‘Auditing behaviour change programmes’	Evaluation	Explains what auditors look for in when assessing the design and evaluation of a behavioural change intervention.
IPA (2016), ‘Resources and Tools for Impact Evaluation’	Evaluation	Beginning with the need for a theory-driven evaluation, and ending with a set of concrete tools to use in running an evaluation, this site provides references to a range of practical materials useful for organizations that are considering a rigorous impact evaluation.

4.3 Further reading – case studies

Document	Topics covered	Notes
FCA (2015), ‘Making current account switching easier – The effectiveness of the Current Account Switch Service (CASS) and evidence on account number portability’ (PDF, 929 KB)	Evaluation, switching remedies, banking.	-
Ofgem (2015), ‘Analysis of the drivers of tariff comparability and clarity of the annual summary’ (PDF, 823 KB)	Evaluation, econometrics, information remedies, restricting choice, comparison tools, energy.	As part of implementing reforms to the retail energy market in 2013/14, Ofgem designed an evaluation framework to go beyond straightforward monitoring of market changes. This report presents the results of this evaluation on Ofgem’s reforms to improve tariff comparability and annual summaries.
CMA (2016), ‘Evaluation of the OFT’s 2012 consumer enforcement action in relation to health and fitness contract terms’	Evaluation, surveys.	Evaluates the outcomes of a consumer enforcement case on improved cancellation rights and reducing the length of minimum contract terms. This is similar to an evaluation of consumer remedies and is a useful example of bringing together evidence from a number of sources including information from competitors and a customer survey.
CC (2013), ‘Understanding past market investigation remedies - Home Credit’ (PDF, 351 KB)	Evaluation, information remedies, search remedies.	Evaluates the remedies on the CC’s home credit market investigation. It finds that there have been few, if any, unintended consequences from the CC’s remedies and that profit per customer of the largest provider (Provident) fell. However, it also found that the CC’s remedies have not necessarily removed all the features the CC identified in 2006.
CC (2011), ‘Understanding past market investigation remedies - Store Cards’ (PDF, 226 KB)	Evaluation, econometrics, information remedies.	Highlights the potential difficulties in data gathering and in isolating with precision the impact of consumer information remedies against changes to other drivers of consumer behaviour. Appendix B of the report applies the ‘difference-in-differences’ approach to evaluate the introduction of APR warnings to sub-set of store cards. Also see this related econometric study: Elliott and Wei (2010), The Impact of Regulatory Intervention in the UK Store Card Industry, International Journal of the Economics of Business, Volume 17, Issue 1, pages 87-88. An earlier version of this paper is available.
OFT (2011), ‘Evaluating the financial benefits for consumers of consumer education and awareness campaigns Just tick it and Skilled to go’ (PDF, 389 KB)	Evaluation, surveys, consumer education, information remedies.	Evaluates the OFT’s Just tick it campaign to raise consumer awareness of scam ticket websites and Skilled to go toolkit for teachers, which aimed to increase consumer knowledge, skills and confidence. Includes a discussion of methodological approaches and the value of early discussions on the evaluation framework before implementing a remedy.
OFT (2008), ‘Evaluating the impact of the Supply of Extended Warranties on Domestic Electrical Goods Order 2005’ (PDF, 306 KB)	Evaluation, surveys, mystery shopping, information remedies.	An independent evaluation of changes to the extended warranties market after the 2005 Order, drawing on interviews, consumer surveys and a mystery shopping exercise (following an OFT market study and CC enquiry). Employed a before and after comparison on the impact of consumer information remedies and a remedy giving consumers the right to cancel an extended warranty.