Research and analysis

How DVSA carries out fleet compliance checks

Published 16 April 2024

Applies to England, Scotland and Wales

1. Introduction

Fleet compliance checks have been run by the Driver and Vehicle Standards Agency (DVSA) or its predecessors (the Vehicle Inspectorate and VOSA) since 1997. Over the years, there have been several changes to the collection, presentation and analysis of the data.

This document aims to lay out the methodology of the current process and to comment upon the previous methodologies.

2. Fundamentals

DVSA checks are normally targeted, both on the locations where offenders are most likely to be found and on the vehicles thought most likely to be offending. Therefore, figures emerging from normal DVSA checks do not necessarily provide a picture of traffic offences and roadworthiness defects that is representative of the fleet as a whole. A fleet compliance check, using random locations and vehicles, is needed to obtain unbiased data.

2.1 Selection of check sites and the sampling framework

Overall approach

In order to combine both condition and compliance checks, the best approach is to sample vehicles at the roadside. This is possible for heavy goods vehicles (HGVs), and the sampling framework is designed on this basis. However, for public service vehicles (PSVs) it is more difficult, as roadside checks would inconvenience any passengers on board. Therefore, the PSV check is split into 2 checks. One focuses on vehicle condition and one on traffic compliance, with the vehicle condition check carried out in depots, and the traffic compliance check carried out at the roadside. The checks are currently carried out in alternate years.

PSV compliance checks are normally performed on coaches, school buses and service buses in the same year, although these can be more focussed on one type, as in 2013 to 2014 when only coaches were checked.

Heavy goods vehicles (HGVs)

In order to produce a representative sample of HGVs, it is necessary to conduct the checks at a diverse range of sites, in terms of geographical location. The times and days of the checks are selected proportionate to HGV traffic. Due to lack of available data, non-Great Britain (GB) vehicles were assumed to follow the same driving pattern throughout the week as Great Britain (GB) vehicles.

All non-GB vehicles in the country had to pass through a port or the Eurostar terminal, many of which had DVSA check sites, so a large proportion of non-GB HGV checks were at or near these locations.

Vehicles passing any given check site were randomly selected for inclusion in the survey.

Public service ehicles (PSVs)

PSVs are checked differently to HGVs. The checks are performed annually with condition and compliance checks being performed in alternate years.

PSV condition checks are performed at operator premises. The Department for Transport (DfT)’s In House Analytical Consultancy provides a sample of operators to the examiners, stratified by fleet size. Examiners will then visit the sites and randomly select the vehicles to check from those they encounter at the site. This is because a full vehicle check can be time consuming, and is impractical to perform whilst the vehicle is in service.

PSV compliance checks are performed at locations where vehicles are expected to be stationary, for example at bus stations or tourist attractions. DfT provides examiners with a number of checks to perform by service type, but do not specify the time, date or location.

Process

DfT produces the random samples for DVSA examiners to carry out checks. During or shortly after each check, examiners complete a data return, which they send back to DfT. This data return varies in format between years, but its key purpose is to identify which checks were checked at random as part of the fleet compliance check. When entering details on the system, examiners should mark the check as a ‘special return’, though this is not always completed accurately. DfT gets an extract of data from this system and match it to the data returns to get a complete list of checks together with outcome.

2.2 Sample size

The sample size required is dependent on a number of factors including the number of vehicles sampled at a single site - this number is known as the cluster size.

There is a chance that there will be a correlation between vehicles examined at any given check site (for example if the site is near a port there will be a disproportionately high number of vehicles travelling by ferry) or at any given operator (where all vehicles are likely to be maintained to similar standards).

As a result, the sample may not be completely random, and a larger sample size is required to compensate for this effect.

The sample size needed for a given level of accuracy, or the level of accuracy from a given sample is determined by:

• the offence rate being measured - the proportion of vehicles with prohibitions, using the previous year’s results as a guide to estimate the current year’s required sample size
• the size of the clusters - the number of checks conducted at each site or operator
• the intra-cluster correlation coefficient, which measures the degree of similarity between measurements made at the same site or operator)

2.3 Measures of offence rates

There are 4 potential measures for compliance with regulations and roadworthiness, which are the proportion of:

• operators committing offences
• the fleet in which offences were found
• journeys made by vehicles which are committing offences
• miles travelled by vehicles where offences are being committed

A case could be made for each of these distinct measures being thought of as providing the best picture. However, there are practical problems in obtaining some measures, and the decision as to which measure to use must therefore take account of the practicalities of carrying out the check, as the different measures would require different survey methods. It is also important to try to determine which measure best deals with issues such as upholding the law, fair competition, and road safety.

For this, and previous GB HGV surveys, the level of compliance by miles travelled by goods vehicles was used. This was partly because it is the simplest of the 4 options to measure, but also because it was felt that it provided the best coverage of the issues involved in HGV compliance of domestic vehicles. For example, regulations could affect different industry sectors differently, and behaviour could change throughout the week.

For this and previous non-GB HGV surveys, the necessary data on miles travelled by non-GB HGVs on different road types were not available. The best data available provided the proportion of HGV traffic which was non-GB vehicles by traffic area - this was used to create the non-GB HGV sample, and aimed to produce a measure of the level of compliance by miles travelled (like the GB HGV measure).

However, due to practicalities, in some areas the checks were carried out at or near entry points to Great Britain, and so each non-GB HGV entering Britain has the same chance of being stopped regardless of how many miles it travels in Britain. This means each vehicle journey had an equal chance of being checked. Therefore, the reported non-GB is not a pure measure of miles travelled by vehicles where offences are committed, but is the best measure considering the data available and practical constraints.

To understand the difference between these measures it is best to consider 2 HGVs both making one journey, but the journey is twice as long for one vehicle. In the GB fleet compliance check the vehicle doing twice the mileage has twice as much chance of being stopped, and this is true for the majority of the non-GB fleet compliance checks. However, as some non-GB checks were carried out at or near an entry point to Great Britain these 2 vehicles would have the same chance of being stopped at those checks, regardless of how many miles they travel in Britain. This means each vehicle journey had an equal chance of being checked.

For PSV compliance, the measure used is closest to the miles travelled measure. However, due to the practicalities of the checks (not inconveniencing passengers), this is an imperfect description.

It is not possible to convert the results of one measure into that of another.

3. Changes to fleet compliance checks over time

3.1 Great Britain heavy goods vehicles (HGVs)

1997 to 2000

GB HGV checks between 1997 and 2000 were performed annually, for an intensive 6 week period.

The check sites were specified in advance by DfT. The examiners would perform large numbers of checks at the specified sites on the specified dates, and complete paper forms which would then be manually entered into a database.

Between 1997 and 1998, the only sanctions for traffic offences were report for prosecutions, but advisory letters were included in 1999.

Factor analysis was largely univariate chi tests.

2001 to 2005

The checks between 2001 and 2005 had a very similar methodology to those in previous years, although the analysis was performed by an external consultancy, Hartley McMaster, and checked by DfT’s Operational Research Unit.

In 2003, new sanctions were introduced, including prohibitions and offence rectification notices. These were grouped with advisory letters in the analysis.

2006 to 2008

GB HGV checks were carried out in 2006 and 2008. The methodology was as before, but the analysis was performed by DfT’s Operational Research Unit, known as In House Analytical Consultancy as of 2008.

2010 to 2015

In 2010, the checks began to be performed throughout the year, rather than in one intensive period. This reduced the total number of checks required. The check sites were no longer specified, instead each area was given a sample split by weekday, time period and road type and asked to choose appropriate check sites.

Instead of paper returns, the data was collected using Microsoft Excel forms, which were emailed to a central mailbox and automatically processed.

In 2010, Graduated fixed penalty deposits were introduced. These were analysed with all other serious offences in a single category.

Analysis was largely multivariate, using logistic regression for the factor analysis. From 2015, it was also used for the defect rate trends and Poisson regression was used of the number of defects trend.

2016 to 2018

From 2016, the samples have been restructured to allow examiners to meet overall quotas based on time and day, but without specifying time and date pairings.

The analysis has also been developed, as detailed in this report.

Trend analysis

In recent years, trend analysis has been performed using non-linear regression (logistic for the prohibition rate trend and Poisson for the number of defects trend). Data is initially cleaned and checked, and then a model is built. If the year coefficient is found to be statistically significant, then there is a trend.

Factor analysis

A new methodology has been devised to analyse the characteristics associated with non-compliance.

1. All checks with any unknown characteristics are removed. If certain characteristics have a lot of unknowns, that characteristic is not considered, and the unknowns in this field are considered.

2. The data is checked to ensure that there are sufficient checks in each category for analysis to be performed. If not, categories are merged.

3. The data is bootstrapped 500 times and the most important variables are found. This requires some decision making on the part of the analyst. As a rough rule of thumb, only variables which are significant in at least 70% of the models are considered, but this may vary if necessary.

4. A final model is built using only the most important characteristics, and the significant results are reported upon.

3.2 Non-Great Britain heavy goods vehicles (HGVs)

Non-GB HGV checks began in 2004 and were repeated in 2006 and 2008. From 2010 they were performed annually.

The analysis performed largely matches that performed for GB HGVs.

In 2014, a road user levy was introduced for non-GB HGVs, which meant that there was an additional offence of not paying the levy. This has been included with all other traffic offences, but some analysis has been performed to identify what effect this has had.

3.3 Public service vehicles (PSVs)

Compliance

PSV compliance checks have been performed between 1998 and 2006. They were resumed in 2009 to be run every two years.

The methodology has largely been the same as the current checks, with the following exceptions:

• until 2011, checks were run for a concentrated period of the year, usually a month or six weeks, which meant that more checks were required
• prior to the 2011 check, results were analysed using univariate chi tests rather than the multivariate logistic regression that is currently used
• in 2013, only coaches were checked, rather than all 3 service types, in order to provide a more focussed picture

3.4 Other check types

Car and light goods vehicle checks have been performed historically, but these were last performed in 2009 and there is currently little plan to continue them. A one-off land train check was performed in 2009.