Research and analysis

Active Travel Fund tranche 2: government response and peer review

Published 9 January 2025

The Department for Transport (DfT) commissioned 2 independent evaluation experts, as members of the Evaluation and trial advice panel, to conduct one peer review of the Active Travel Fund (ATF) stage 1 process evaluation report, and one peer review of the Active Travel Fund: evaluation baseline report.

DfT is grateful to the peer reviewers, and we welcome their findings. Both peer reviewers have endorsed the approach taken in the ATF evaluation. Where possible, we will put recommendations into practice to maximise the learnings from the ATF evaluation.

ATF evaluation baseline report

The peer review confirms that the ATF evaluation uses best practice methods across the four themes, taking care to separate intervention and comparison areas in such a way that secular changes can be distinguished from those due to the intervention. There is good alignment between the evaluation questions, the evaluation approach, and the methods used, following best practice guidance as set out in the magenta book.

The peer review additionally confirms that, where data challenges lie, the evaluation team have made rational choices in view of resources with appropriate data handling methods and optimal analytical approaches.

In response to the peer review feedback, DfT has implemented the following updates to the report:

• a worked example to demonstrate how desire lines were selected (research question (RQ)2)
• text to describe how the evaluation defines boundary roads (RQ4)
• text on how school term has been considered (RQ4)
• text to provide the rationale for the free school meals measure for the school streets theme (RQ8)
• the removal of RQ9 (which brought together the findings from the previous RQs)

DfT acknowledges the following limitations identified in the report and aims to learn from these for future evaluation activity.

RQ 4

The peer review notes that possible harms seem to be glossed over. Given the limited resources available, we acknowledge that it has not been possible to measure the impact of all possible harms. For instance, if the research included a longitudinal panel study of change in individual travel behaviour, it would be possible to compare women’s and men’s changes in behaviour and attitudes to see if one gender benefits more than the other, but the research team cannot do this here as such a study was out of scope and budget.

The research team have looked at the important possible disbenefits related to car journey times and increased congestion on boundary roads, and we will investigate as previously if there are any increases in injuries on boundary roads. We acknowledge these do not represent all possible harms.

RQ 5

The peer review notes that the analysis lacks statistical power for this research question. DfT accepts that the sample is small and that the data will be in its early stages, however, there is still value in reporting the information that is gathered, even if it is not possible to draw robust conclusions.

RQ 7

The peer review notes the usefulness of reporting on local authority stakeholder diversity. Organisational diversity is reported in the ATF process evaluation stage 1 report.

Demographic data was not collected on respondents who represent local authorities rather than specific segments of the population.

Peer review recommendations

The peer review recommends the removal of material that does not have real power to tell readers what works for whom and in what circumstances.

DfT acknowledges the importance of this but notes that this is a baseline report, final reporting will be focused on what works for whom, with conclusions being drawn only where there is sufficient statistical power.

The peer review recommends developing a theory of change for each of the themes to better understand the mechanisms of separate active travel interventions.

DfT welcomes this suggestion for future evaluation activity but is out of scope for this evaluation due to limited resources.

ATF stage 1 process evaluation report

The peer review confirms that the stage one process evaluation is based on robust analysis, with logical conclusions based on the themes and analysis considered throughout the report.

Furthermore, DfT is pleased to learn that the peer review identifies the process evaluation as representing considerable value for money, which meets its objectives and draws out important, balanced, and practical learning points for the future.

DfT acknowledges the limitations identified in this report and aims to learn from them for future process evaluation activities. In response to the peer review feedback, DfT has implemented the following updates to the report:

• information on research ethics
• text to describe the internal validity of the thematic analysis approach
• text to better summarise the limitations of the study

DfT acknowledges the following limitations identified in the report and aims to learn from these for future evaluation activity:

• the peer review notes that the results of the report could have been grouped more effectively together. DfT welcomes this suggestion for future reports, noting that the decision to structure the report around the RQs followed a previously agreed approach
• the peer review highlights the value of supplementary research methods such as participatory approaches with the local authority staff participants. DfT welcomes this suggestion for future process evaluations, noting that this was out of scope within the available resource
• the peer review states that more information on resolving challenges would have been useful, including lessons learnt that are relevant for both national and local areas. DfT notes a few examples provided in the report where local authorities describe resolving challenges, acknowledging that identifying unresolved challenges was not unexpected as this is a first stage process evaluation considering early scheme implementation
• the peer review suggests that the wider literature should be embedded in the results to understand how research findings fit into the wider context. This was considered out of scope given the available resources, DfT confirms that this does not make the approach less valid

ATF evaluation baseline report: peer review statement summary

This report by the Local Transport and Regional Analysis Consortium evaluated the baseline data of the ATF by way of a thematic approach.

It uses best practice methods across the 4 themes that it considers using quasi experimental approaches including ‘difference-in-differences’ and ‘interrupted time series’ methods as well as more conventional ones. They have incorporated secondary data, new data collected by local authorities and some qualitative data.

The methods used take care to separate intervention and comparison areas in such a way that secular changes can be distinguished from those due to the intervention alone. This aims to allow the research to assess health, social and economic changes due to the interventions rather than other things.

While it was good to see a clear ‘logic map’ presented, it would have been helpful if they had a theory of change developed for each of the themes/interventions so as to better understand the mechanisms of each of them. This was most clearly evident in the case of the ‘school streets’ theme where the mediators of effect might well have to do with volume of traffic or size of school and yet currently the analysis concerns free school meals as a proxy for deprivation.

While looking at protected characteristics and having a ‘what works for whom’ approach is to be applauded, it is unclear why this might affect the power of the intervention.

In some themes, there were problems with the absence of data (such as the LTN study) and the investigators have had to make decisions about how to handle this issue. By and large, they have made rational choices in view of their resources and data handling/cleaning methods were all appropriate. Further, the interrupted time series and controlled interrupted time series approach is, I believe, the optimal approach for this theme.

Within the existing budget, I would suggest that the RQs be reconsidered for clarity and a question that is something like ‘what evidence of harm or unintended consequences from any of these interventions has been indicated?’ be presented across the themes.

RQ9 should, I believe, be totally rewritten as it brings together far too many different unrelated issues and is really a summary section for the report rather than a RQ and as such should be thought through with greater care.

Were a larger budget to be made available, I would suggest that proper theory of change exercises be done for each theme which should involve stakeholders of all kinds to ensure that the outcomes assessed and mechanisms are better understood.

This may enable the team to tweak their assessments and the data that they pull down from secondary sources. This will also enable some issues of selection bias to be considered across the different themes.

ATF evaluation baseline report: full peer review

Materials and methods

This report presents the baseline evaluation of the ATF tranche 2 with elements of the emergency ATF1 by the Local Transport and Regional Analysis consortium, led by the University of Westminster. It takes a mixed methods approach to answering 9 RQs which are based thematically. Most of the data reported here are quantitative and, by and large, automated. This approach aims to compare the various intervention and comparison areas in such a way that secular changes can be separated from those due to the intervention alone.

This aims to allow the research to distinguish health, social and economic changes due to the interventions, from those that might be due to, for example, those made by Instagram influencers on cycling.

The 9 RQs are themselves ambitious, particularly when the budget set for the research work overall is borne in mind. Firstly, it was good to see that there was an overall logic map for the project, although I thought this could have been used better. Indeed, each intervention (or theme at least) would have benefitted from having a clear theory of change shown to indicate what the proposed mechanism might be lying behind it. With that theoretical mechanism agreed, the researchers would be better placed to consider in future how to consider effects (or lack of them) downstream. Currently, what exists will not provide them with such a clear mechanism that stakeholders have been part of to agree on what is going on across each intervention.

A clear example of this is shown in school streets (theme 4/RQ 8) – we are told that this intervention concerns restricting traffic around streets near schools at key times. While there is some background evidence in a possible theory of change around NO2 levels and active travel from the Davis review (2020). In line with magenta book guidance, the team went on to look at schools in disadvantaged neighbourhoods (measured by free school meals) as a priority.

This takes a ‘what works for whom approach’ and is, in general, to be applauded. It is not clear however why this decision was made. I wondered why greater focus was not given to larger schools versus smaller ones? Or schools with busier roads around them versus quieter roads? Secondary versus primary?

While a diversity approach is important, particularly around progress plus dimensions, I wondered whether this is really the mediating issue that matters here or how it was decided upon. A clear theory of change for each intervention might enable the team to better explain their decisions.

Secondly, some questions appear to ask similar items, for example, RQ 5, RQ 8 and especially RQ 9 all ask about road traffic casualties/injuries. This is a point of particular interest to readers of a report such as this and in line with advice in the magenta book.

I would suggest that the RQs be reconsidered for clarity and a question that is something like ‘What evidence of harm or unintended consequences from any of these interventions has been indicated?’ be presented across the themes. RQ 9 should, I believe, be totally rewritten as it brings together far too many different unrelated issues and is really a summary section for the report rather than an RQ and as such should be thought through with greater care.

In general, however, most RQs and their associated methods, are appropriate. I was impressed by the reporting of the methodology in RQs 1, 2 and 3 and how the importance of standardisation of measures was explained. However, in RQ 4, the team might better address the issue of boundary roads and possible selection bias by explaining how this process was done. Possible harms within RQ 4 seem to be glossed over.

Within RQ 5, the analysis of car ownership lacks power. Were a priori analyses conducted that gave confidence for this work to be conducted? It seemed to contain such little data of value that it may be better to withdraw it.

RQ 7 might have better reported the diversity of the local authority stakeholders. For such work, it would give better credibility to the data if readers were reassured with better information on the participants.

Overall, I found good alignment between the evaluation questions, the evaluation approach and the methods used. As with most complex interventions, evaluation designs typically involve mixed methods, combining both qualitative and quantitative methods to answer the impact, process and value-for-money questions and the team follows magenta book methods well using difference-in-differences and interrupted time series methods.

Time and resources were limited, questions and methods had to be prioritised and trade-offs between methods were inevitable. In my opinion, they selected feasible, appropriate methods in view of the costs, ethics and response rates that were manageable in the time available.

Analyses and results

These were both generally conducted well and in line with the plans proposed earlier in the report. For the budget the team had to work with, I am impressed with the considerable amount of work that has been achieved.

Regarding cycle track schemes (RQs 1 and 3), data were collected on cycling flows pre-construction and trends reported via counts. Care was taken to get consistent data and to consider context (including the COVID-19 pandemic) as best they could.

Reasonable assumptions regarding missing data were made and the paired count approach is optimal to assess causality. While many of these counters cannot separate pedestrians from cyclists, with the resources available this limitation is something they can deal with. The difference-in-differences approach is entirely sensible and was well planned and reported.

The LTNs presented the team with considerable challenges where data were often lacking or were hard to match. There were also 2 local authorities that did not collect automated data at all as well as others who dropped out of the scheme. I was not entirely convinced that selection bias was well managed in this theme (RQ2) although the final table on matching was impressive on the variables listed matters such as how desire lines were selected are not clear.

Data handling/cleaning methods were all appropriate and the Interrupted time series and controlled interrupted time series approach is, I believe, the optimal approach for this theme.

I was impressed with how the team was thoughtful about context, such as consideration of time of day and day of the week in its analyses such as in its work on RQ 4 (boundary roads in the LTN). However, they did not seem to look at the school year, which seemed like an odd oversight. While the work is novel, interesting and a well conducted difference-in-differences analysis, in view of the lack of generalisability of this question I wondered whether this might be removed from the report or pushed to an appendix.

The analysis of the Driver and Vehicle Licensing Agency (DVLA) data on car ownership had considerable intuitive appeal but a lack of statistical power. This was surely entirely predictable and I wondered why time had been spent on this work. I would suggest a simple statement to this effect rather than appearing to have spent time on a rather fruitless task.

The Road and Transport Authority (RTA) data on LTNs appears to suffer from floor effects whether ‘casualties’ or ‘killed’ and ‘seriously injured’ categories. While this outcome too is underpowered, the detection of harm such as this is likely so important to readers that I would propose that it remains in the report. These data are strengthened by the source being from stats19 and the relative simplicity of the analyses.

Similarly, the school streets theme begins with analyses of RTAs within 100 to 200 metres of schools based on the same data source. The analytic proposal here seems sound and again, even though power is low, the outcome is likely so important that even indicative data will be acceptable.

However, the analysis of ‘school streets’ projects compared with children in receipt of free school meals seems very odd to me as I fail to see the connection via the (missing) theory of change. While effects may vary by economic group, this is surely not the most significant mediator in these cases.

Reporting has been generally done well in a rigorous and careful way. The team has been thoughtful about generalisability, for example carefully disaggregating data about London and non-London schemes. Comparability between intervention and comparison sites has manifestly been challenging across this project.

However, by and large, the team has done remarkably well to achieve what they have in view of the resources available to them. While there may well be criticism that some counts (of transport modes for example in LTNs) vary considerably – I think that the points they aim to make are still well made. They have been cautious to consider the varying types and qualities of data carefully, for example, separating the data from Oxford and Birmingham in the LTN analyses. This is entirely appropriate. Also, combining data where possible such as that from RUIS makes good use of what is high quality available data and allows for robust, helpful analyses.

Conclusions

Overall, this is an ambitious piece of work reporting a large amount of data from a small budget. That said, the team has done extremely well and is to be congratulated. It may help them to report less and to cut out the material that does not have real power to tell readers what works for whom in what circumstances.

Much of the data is, for various reasons, limited and they should simply remove those sections. Clarity about the mechanisms by having putative theories of change for each intervention could be helpfully added to allow readers to better understand what is going on in each theme and to assess generalizability.

ATF process evaluation stage 1 report: peer review statement

Overall, the evaluation report is very well written, generally coherent and based on robust analysis.  The researchers provide a broad overview of the issues and challenges faced by areas and also the learnings that can be drawn for the future. There is also the reporting of successes and practices that might provide government and local areas with ideas going forward, for example, the importance of local stakeholder engagement and how research can feature as part of such processes.

Generally, the learnings identified for local and national policymakers are logical based on the themes and analysis considered throughout the report.

In terms of areas for reflection, I wonder if the researchers have drawn enough from the data? For example, when discussing the challenges faced by areas was there more data on how to resolve these challenges, for example, local skills capacity issues, multi-level governance issues, funding shortages experienced locally, how to ensure more of a representative sample of participants within local consultations on the schemes?

It might genuinely be the case that the data is not there, but it was surprising to not see more of this given the nature of the participants and their positions, etc. Moreover, including lessons that crosscut and are relevant for both national and local areas is an area that could be strengthened.

I do think an aspect for reflection within the existing budget is that there could perhaps have been more dovetailing in the report with how the findings are consistent or otherwise with the extant literature on the implementation of active travel schemes/health interventions, especially given the expertise of the team.

It might be the case that the researchers feel that taking a robust approach to embedding literature in the results would be more possible if more funding was available. For example, the process evaluation could have been supported by investment in a systematic review of similar schemes, perhaps from overseas, that could have shaped the focus group discussion questions to allow for broader analysis and reflection on the implementation of schemes.

If more funding was available then the researchers could have unearthed the lived experiences of the implementers in local areas and placed these experiences within a broader context.  Indeed, researchers could have used ethnographic methods within areas and participatory approaches within the process evaluation. This would have brought a strong degree of nuance and insight into local planning and delivery.

That said, and with the available budget in mind, I believe that this current process evaluation represents considerable value for money. In short, this this a strong process evaluation that not only meets its objectives but also draws out important, balanced and practical learning points for the future.

ATF process evaluation stage 1 report: full peer review

Materials and methods

The methodological approach was appropriate in terms of qualitative focus group interviews. The sampling strategy is also appropriate with no bias detected. The research design is described sufficiently, and the evaluators provide confidence that the focus groups were structured effectively and underpinned by appropriate questions as per the nature of the evaluation.

There is no mention of research ethics and how this was secured. This needs to be clarified prior to publication given the involvement of human subjects. This, unfortunately, is not in line with the principles of the magenta book. However, it appears that all other aspects of the evaluation report are in line with the principles.

Analysis

The researchers discuss using thematic approaches to the analysis based on the focus group questions. There could have been more detail provided around the internal validation of the dominant themes that emerged as well as the ‘other factors’ covered in the report.

For example, were focus groups analysed by one person or more than one person?  If more than one person what was the process and protocols for agreeing on the key issues and quotes that are identified to feature in the report? In short, more detail regarding maximising the validity of the analysis would be welcome.

The analysis of the data shows the richness of the focus group data.  It was very surprising (and this is a general comment about the report) to not see more linkages made between the findings and trends in the active travel and public policy intervention literature.

A strong example here is the nature of consultations/engagement and how youth engagement is more difficult – this is in line with the wider literature. Also, issues with monitoring and evaluation, challenges of short-term funding cycles, the importance of local political leadership support, and the consequences of not identifying national and local success criteria from the outset are not exclusive to this evaluation (so many evaluations across many policy sectors warn of the dangers of failing to do this).

The evaluation is, of course, a focused piece of work that is driven by key questions but the value of evaluations is not always only about the precision of exploring particular questions, it is also about how the findings around questions can be placed within a wider context.

Results

The results are written up well and the evaluation provides learning for national and local policymakers. The results are likely to be useful for future policy conversations and for the DfT and local planning.

There are areas of the report where the results could have, in structural terms, been grouped more effectively together. For example, capacity issues and skills gaps within areas in terms of implementation were major themes. However, this was spread across the report. There is a strong case for capacity issues to be grouped together more effectively, for example, this theme appears in main sections and under ‘other factors’, which read as slightly confusing.

In terms of the presentation of the data, there were two main areas where the results did not follow through into the learning section for national and local policymakers, and these are learnings which cut across both levels.  For example, the focus group results cover issues with multi-level governance and skills gaps on monitoring and evaluation and being about to join up activities to meet objectives of the active travel interventions. 

Therefore, I suggest another section could be added that draws about learning for both types of policymakers – one that recommends that the challenges of multi-level governance tensions are resolved (or attempted to be resolved) through co-creation on the planning and evaluation of the local schemes with multi-level stakeholders (including representatives from the DfT) and another that focuses on how capacity development around the local skills on M&E and delivery need to be part of future conversations between local and national stakeholders.

It would also be useful to know from the data if focus group members had reported how to resolve several of the challenges identified through the evaluation – the report seems to focus more on describing the challenges themselves.

Conclusions

See comments in the results section above. However, the conclusions are reasoned and based on a robust evaluation. The conclusions drawn from this report are based on the deployment of an appropriate methodological approach and sound analysis. The conclusions are clear.

The report is not clear on the limitations of the study and more should be said about this.