Gambling Act Review evaluation plan
How DCMS and the Gambling Commission are working together to deliver an appropriate programme of work to evaluate the impact of the Gambling Act Review.
Foreword
The Department for Culture, Media and Sport (DCMS) and Gambling Commission (GC) procured The National Centre for Social Research (NatCen) to undertake an evaluation scoping study for the Gambling Act Review (GAR) in January 2024, and subsequently to deliver that evaluation. The gambling white paper, published in April 2023, sets out the previous government’s plans for reform of gambling regulation following the Review.
The gambling white paper contains over 60 proposed policy measures, to be delivered by a mixture of different stakeholders, making it a highly complex policy initiative. The policy measures have different timeframes and implementation processes, and potentially intersecting outcomes and impacts. Attribution or contribution of observed changes to any one policy measure is therefore difficult.
It is, however, important that policy measures implemented as part of the GAR are evaluated, despite the challenges inherent in doing so. DCMS and the Gambling Commission are working together to deliver an appropriate programme of work to evaluate the impact of the policy measures implemented following the GAR. The evaluation will be responsive to live policy decisions to ensure that it answers the evaluation questions with respect to final policy positions. The timing of evaluation activities have been extended in this context with further decisions to be made on the detailed scope and approach of the evaluation.
In the meantime, the below outline of the GAR evaluation plan presents the underlying principles and approach to the evaluation of GAR including evaluation aims and objectives, evaluation questions, the analytical framework, and research methods.
The outline of the GAR evaluation plan provides details of an evaluation advisory group and lived experience panel, which are important in making the evaluation representative and robust. In addition, DCMS and GC will also continue to engage with stakeholders on specific evaluation activities.
Elements of evaluation fieldwork are planned to begin in January 2025.
Introduction
In December 2020, the Department for Culture, Media and Sport (DCMS) undertook a review of the Gambling Act 2005. The Gambling Act Review (GAR) examined whether the 2005 Act provided the right “balance of regulation” in the digital age. The Review’s objectives were to:
- examine whether changes are needed to the system of gambling regulation in Great Britain to reflect changes to the gambling landscape since 2005, particularly due to technological advances;
- ensure there is an appropriate balance between consumer freedoms and choice on the one hand, and prevention of harm to vulnerable groups and wider communities on the other; and
- ensure customers are suitably protected whenever and wherever they are gambling, and that there is an equitable approach to the regulation of the online and the land-based industries.
The gambling white paper, published in April 2023, sets out the previous government’s plans for reform of gambling regulation and covers 62 policy measures. Responsibility for taking these forwards is split across DCMS and other government departments using legislation; the Gambling Commission (GC) using its powers under the Gambling Act 2005; and the gambling and related sectors taking voluntary action.
In January 2024, DCMS and the GC commissioned the National Centre for Social Research (NatCen) to undertake an evaluation scoping study. The scoping study held a series of Theory of Change (ToC) workshops with DCMS and GC colleagues to identify a series of specific policy measures that had been developed since the publication of the white paper. The scoping work highlighted that this evaluation plan requires a proportionate approach, focusing on a number of specific policy measures in detail.
NatCen began work in the summer of 2024 to deliver the evaluation.
Evaluation aims and objectives
The evaluation will focus on evaluating the impact that can be attributed to a number of specific policy measures implemented under the Gambling Act Review. It will seek to establish, as far as possible, whether in-scope policy measures are being delivered effectively, understand if they are achieving their intended outcomes, impacts, any unintended consequences, and capture learning to inform the design and implementation of any future policy changes. The impacts and outcomes of specific proposals will be evaluated individually, as well in a package to understand the collective impact of policy measures that have been implemented so far and can be evaluated within the lifetime of this evaluation.
The policy measures in focus of this evaluation plan are many and diverse and the implementation context is complex, dynamic and evolving. Following the General Election in July 2024 and change of government, the evaluation will be responsive to live policy decisions to ensure that it answers the evaluation questions with respect to final policy positions. The timing of some evaluation activities have therefore been extended in this context, with further decisions to be made on the detailed scope and approach of the evaluation. The ToCs developed for the evaluation plan in the scoping study indicate that the observed impacts are likely to result from combinations of multiple causes with the intervention of multiple GAR policy measures contributing alongside other external factors. This is why it is important that this evaluation aims to establish plausible causation related to the GAR policy measures with a high degree of certainty, carefully factoring in the impact of alternative explanations.
Evaluation questions
In line with the aims and objectives set out above, the evaluation will aim to gather evidence and formulate findings against the following research questions. The list below includes a mix of impact (Q1, Q3-Q6) and process (Q2, Q6, Q7) questions that will guide the respective evaluation strands.
1. What intended and unintended outcomes and impacts have been brought about in the short and longer term by the GC, DCMS and independently led proposals in the two-year evaluation that were introduced following the GAR?
- 1.1 In what ways, if any, did these outcomes and impacts link and interact?
2. Were the proposals implemented as intended?
- 2.1. Did the proposals introduced reach online and land-based gambling providers and consumers as intended?
- 2.2. In what ways, if any, did implementation vary from that intended and why? If there was variation, what were the associated impacts?
3. What was the distinct contribution of the DCMS, GC and independently led proposals in achieving the observed outcomes and impacts? Specifically….
- 3.1 how and why did the proposals contribute to reducing gambling-related harm to vulnerable groups and their wider communities in the short-term and longer-term (or not)? What worked best for whom, why and when?
- 3.2 how and why did the proposals contribute to increasing gambling protections, while ensuring a fair and open safeguarding of gambling-related consumer freedoms and choice for customers in the short-term and longer-term (or not)? What worked best for whom, why and when?
- 3.2 how and why did the proposals contribute to regulating the online and the land-based gambling industries more equitably in the short-term and longer-term (or not)? What worked best for whom, why and when for the online and the land-based industries?
4. What was the combined contribution of the proposals in achieving the above observed outcomes and impacts?
5. What conditions were necessary for the proposals to achieve the above observed outcomes and impacts?
6. What internal and external influencing contextual factors supported or impeded the proposals to achieve the above observed outcomes and impacts? - 6.1. In what ways, if any, did internal and external influencing contextual factors interact with the proposals?
7. What are the implications of the findings from the evaluation for the implementation of future gambling-related policy changes?
Evaluation design
The evaluation is structured around a theory-based design combining Contribution Analysis and Process Tracing. Contribution analysis is a step-by-step approach to data collection, triangulation and analysis based on a ToC and testable causal contribution claims. It will be used to first develop the hypotheses (the causal contribution claims), and the principles of Process Tracing will then ensure that the hypotheses are empirically testable and guide data collection. This will ensure, as far as possible, that conclusive evidence on contribution is gathered, establishing a clear degree of confidence in each claim. The evaluation will draw on multiple sources of evidence, including qualitative insights and findings from quasi-experimental quantitative analysis: the latter will enable robust causal estimates of the degree to which changes in outcomes can be attributed to specific GAR policy measures.
Methodology and data collection
Theory-based approaches to determining causality can leverage any kind of methodology to gather the evidence needed to test claims or hypotheses. To evaluate GAR measures, a comprehensive mixed methods evaluation approach will be required to estimate impacts and otherwise capture whether, how and why observed outcomes came about. Statistical and qualitative impact and process data collection and analysis will be run in tandem. The research methods planned for the evaluation include:
- longitudinal consumer surveys
- operator surveys
- quasi-experimental designs (for example, Interrupted Time Series, Regression discontinuity design, and Difference in Differences approaches
- in depth interviews with operators, consumers and other stakeholders
- focus groups
- diary studies with young people
- tracking and monitoring
Multiple data sources will need to be accessed across the evaluation strands. For quasi-experimental designs, online and land-based operator data will be used. For qualitative impact and process evaluation, methods including in-depth interviews, focus groups, diary studies, and surveys will be used. These would involve a range of participants, including operators, people who gamble, and other relevant stakeholders, such as local providers of support services. These will be complemented by periodical tracking of the implementation of the GAR policy measures and external influencing factors.
Evidence synthesis and reporting
In line with a Contribution Analysis approach, data synthesis will combine process and outcome evidence. This will enable rigorous explanations of how and why observed changes came about (or not), the conditions needed for intended outcomes to be realised, as well as risks and enablers to the pathways to change and the role of external influencing factors.
The presentation of evaluation findings will need to do justice to the richness, diversity and congruence of the qualitative and statistical data. A top-down approach is likely to be most effective given the diversity of evaluation data collection activities. That is, starting from the high-level process and impact research questions to be answered and then looking across all evidence sources for findings that answer more specific questions. Along with the data collection tools and (anonymised) datasets developed/used, several outputs during and at the end of the evaluation will be provided to DCMS and the GC to enable evidence-based insights in ongoing policy development and decision-making.
We expect the evaluation to report in 2026.
Research governance
The evaluation will include a Lived Experience Panel (LEP) bringing together people with lived experience of gambling and gambling harm, working in conjunction with GC’s pre-existing Lived Experience Advisory Panel (LEAP). The panel will aim to provide guidance and input throughout the project, including advice and direction to the evaluation, and ensure that the voices of different groups with lived experience of gambling and gambling harm, including those of affected others, are considered. The panel will be administered by NatCen.
An Evaluation Advisory Group is being set up to provide evaluation expertise to ensure the Supplier has objective, external advice on how best to take the evaluation forward. This will bring together researchers, academics and evaluators with expertise and experience in the field of gambling policy, research and regulation. The group will provide expert advice and input, independent assurance for key evaluation products and outputs, and assist evaluators in the anticipation and mitigation of risks and issues which may impact the evaluation. The group will be administered by NatCen. Neither group play a role in policy development.
This evaluation will be subject to an enhanced approach to research ethics, with clear considerations to be made based on the subject matter and the involvement of potentially vulnerable participants. NatCen will subject the evaluation to a full ethical review that meets the requirements of the Economic and Social Research Council and Government Social Research Professional Guidance – Ethical Assurance for Social Research in Government.