Accredited official statistics

Methodology

Updated 25 July 2024

Total Factor Productivity of the Food Chain

This is a methodological note accompanying the Official Statistics release published on 25th July 2024. The original research that forms the basis of the release was published in May 2006 and is available on the archived Defra website here: UK Food Chain Productivity Incorporating External Impacts.

1. Introduction

Total factor productivity (TFP) relates a change in output to changes in major inputs of labour, intermediate purchases and capital consumption, and is a measure of efficiency. The release presents measures of TFP within the food sector downstream from farming divided into: manufacturing, wholesale, retail and catering. A measure of the TFP of the wider economy is included for comparison purposes.

Productivity growth is linked to the potential competitiveness of companies. Growth in productivity means businesses can meet their commitments, for example to employees, shareholders, and governments, and retain or improve their competitiveness.

There are benefits to society from a rise in productivity as it raises living standards. Productivity is an underlying indicator of sustainable resource use but it cannot be used a single measure of sustainable growth.

2. Description of food chain sectors

In SIC2007 the food manufacturing sector comprises of nine main categories including processing and preserving meat, dairy, fruit and vegetables, oils, bread, biscuits and cakes, and confectionery. Animal feed manufacturing is included, covering both farm animal feed and pet food, and representing 8.2% of total turnover in food and drink manufacturing in 2022. The drink manufacturing sector includes alcoholic beverages and soft drinks and mineral waters.

Food and drink wholesale consists of the buying, storage and reselling of food either manufactured or freshly produced. Wholesale of tobacco products (46.35) is not included, but SIC code 46.17 "Agents involved in the sale of food, beverages and tobacco" is included. This group includes wholesalers that trade on behalf of others on a fee or contract basis and their turnover was 2% of that of 46.3 "Wholesale of food, beverages and tobacco" in 2022.

Food and drink retail is defined as the sale of food within both non-specialised stores (e.g. supermarkets), 47.11, and specialised stores such as butchers and bakers, 47.11 and 47.81. The sale of tobacco products is subtracted from the specialised stores using 47.26 and then subtracted from the non-specialised stores later on using a ratio for food and drink.

Non-residential catering consists of restaurants and bars involved in preparation and serving of food, alongside canteens and catering services. Hotels are not included.

Table 1 Standard Industrial Classification sectors used in the model

Sector	Standard Industrial Classification	SIC2007
Manufacturing	Manufacture of food products Manufacture of beverages	10 11
Wholesale	Wholesale of food, beverages and tobacco minus Wholesale of tobacco products plus Agents involved in the sale of food, beverages and tobacco	46.3 - 46.35 + 46.17
Retail	Retail sale of food, beverages and tobacco in specialised store minus Retail sale of tobacco products in specialised stores plus Retail sale in non-specialised stores with food, beverages or tobacco predominating plus Retail sale via stalls and markets of food, beverages and tobacco products	47.2 - 47.26 + 47.11 + 47.81
Non-residential catering	Food and beverage service activities	56

3. Defining Total Factor Productivity

Total factor productivity is a measure of the efficiency with which inputs are converted into outputs. The method incorporates only the inputs and outputs that are associated with monetary transactions. It does not incorporate external effects on society and the environment. The inputs are labour, capital expenditure and purchases. The output is the volume of sales. Total factor productivity is the 'volume of outputs' divided by 'the volume of inputs'. TFP increases if the volume of outputs increases while the volume of inputs stays the same. It increases if the volume of inputs decreases while the volume of outputs stays the same.

It is not possible to add the volumes of different types of inputs together, but year on year changes in the volumes of different types of inputs can be used. Thus:

Change in total factor productivity = (Change in volume of outputs) / (Change in volume of inputs)

For this reason the exact level of total factor productivity is not calculated but year on year changes in total factor productivity are identified. Thus the measure is an indicator that shows the trends.

Weights used to combine inputs

According to index numbers theory it is appropriate to use the values of costs of the inputs as the weights when forming the weighted average of their trends. The method uses Fisher Indices with annual rebasing and chain-linking to keep the weights up to date. The full description of the indexing procedure used is included in the original research report Page 3.

A weighted average of the trends in the inputs is formed to combine trends in labour, capital consumption and purchases. The weights are the current price (not adjusted for inflation) costs of the three inputs which vary by industry sector as outlined in Table 2.

Table 2: Input weights: the proportions of labour, capital expenditure and purchases per industry sector in 2022

Category	Capital expenditure(a)	Purchases	Labour costs
Manufacturing	0.03	0.82	0.15
Wholesale	0.01	0.91	0.07
Retail	0.03	0.78	0.18
Catering	0.06	0.57	0.36
Wider economy	0.04	0.78	0.17

(a) The weight used for manufacturing is MI3N Manufacture of food, beverages and tobacco from the ONS Capital Stocks series. For all other sectors ABS Total net capital expenditure is used.

For all sectors, purchases have the greatest share of costs so they have the greatest weight when forming the combined trend. Labour costs only account for 7% percent of wholesale inputs but they comprise over a third of catering sector costs. Therefore an increase in labour costs will have less of an overall affect on the wholesale sector than the same increase would have on catering, if all other factors stay the same.

4. Identification of data sources for inputs and outputs

Volumes of high level aggregates of outputs or inputs are not collected directly in surveys because of the impracticality of adding volumes of different types of output or input together. Instead, current price values are the starting point and these are then converted to volumes by removing the effects of price changes. This approach is taken for the volume of output, the volume of purchases and the volume of capital flow. For labour, there are direct estimates of volume in terms of hours worked.

Data is not always available at the SIC levels outlined in Table 1, and some assumptions have had to be made about how trends in for example the wholesale sector as a whole apply to food wholesaling as discussed below.

Table 3 shows the data sources selected for the model. The ABS estimates cover all UK businesses registered for Value Added Tax (VAT) and/or Pay as you Earn (PAYE).

To convert turnover and purchases from price to quantity deflators are used for each sector as outlined in table 4.

Table 3 Data sources for inputs and outputs

	Unit	TFP Calculation Component	Data source
Output
Turnover	£	Total Turnover (Sales of Products)	ABS
	£	Deflated to 2000 prices	Deflators vary by industry sector (see Table 4)
Inputs
Labour	£	Total employment costs	ABS
	Hours worked	Annual Hours Worked adjusted by industry estimates for fulltime and part-time hours	ABS/BRES employment numbers weighted by ASHE
Capital	£	Capital Expenditure	ABS
	£	Consumption of Fixed Capital: by Industry at Chained Volume Measure	ONS Capital Stocks
Purchases	£	Total purchases of goods, materials and services	ABS
	£	Deflated to 2000 prices	Deflators vary by industry sector (see Table 4)

Turnover

Food retail turnover will include sales receipts from non-food retail. Since the introduction of SIC2007 this includes petrol sold at supermarkets alongside clothes, electricals, magazines and newspapers and tobacco. The ABS produce a breakdown of retail turnover (inclusive of VAT) by product category from which it is possible to estimate the proportion of turnover from non-food sales. In 2018 33% of food sector retail came from non-food items. Total turnover from ABS is adjusted to remove the non-food sales. (This split of food and non-food is currently not produced by ONS, therefore 2018 data is used for more recent years too.) Purchases are also adjusted by the same proportion as described below. See also turnover deflators Table 4.

Labour

Numbers of employees from the ABS Survey are converted into hours worked using hours per week estimates from the Annual Survey of Hours and Earnings (ASHE). ASHE gives estimates of numbers of jobs within each SIC but these categories do not exactly match the level of detail available in ABS. Median rates are a weekly average and therefore assumptions need to be made on the number of weeks worked per year to create an annual series. The legal minimum number of weeks allowed for paid leave is used. From 2000 to 2008, four weeks including public holidays is used and for 2009 onwards 5.6 weeks is assumed. Employment estimates ABS are point in time estimates, not seasonally adjusted.

All hours worked are treated as being of the same quality. The skill level of the workforce and changes in skill level are not accounted for and will not affect the resulting productivity measure.

Capital Input

Companies have a stock of capital made up of assets of differing ages but it is not a direct input into the production process. It is the 'flow of physical capital services' which is needed for the total factor productivity measure. The flow is not directly observable but can be estimated using a perpetual inventory model that requires estimated lifetimes and depreciation rates for each type of capital asset. ONS publish datasets on the consumption of fixed capital by industry, but not at the SIC level required by the model apart from food manufacturing (but this includes tobacco).

The rate of change of capital stocks in distribution sector as a whole is assumed to be the same as that for both food wholesaling and food retailing. Hotels and restaurants capital consumption is used for NRC. The chained volume measure is used which provides values in constant prices in respect to a base year.

The weight used for food and drink manufacturing is MI3N Manufacture of food, beverages and tobacco from the ONS Capital Stocks series. For all other sectors Total net capital expenditure values in real terms are taken from ABS Survey at SIC code level to form the weights.

Purchases

The retail sector purchases include non-food purchases. There is no data source that directly measures the proportion of non-food retail purchases. It is assumed that the proportion of purchases that are non-food is the same as the proportion of retail turnover that is made up of non-food sales.

5. Deflators

The effects of changing (usually rising) prices need to be removed from the current price trends. Appropriate price indices are selected to divide in to the current price indices. This is particularly important in the food chain because prices can change at different rates, for example food price inflation (CPIH) lagged behind general inflation for many years until late 2008 where the situation was reversed until 2015.

Table 4: Purchases (inputs) and turnover (output) deflators

Sector	Purchases deflators	Turnover deflators
Food manufacture	PPI for food and drink purchases (MC35, then GHHV from 2011)	PPI output of food products (K37L, then G6SI from 2011).
Food retail	PPI Output of food products (K37L, then G6SI from 2011) and the Agricultural Price Index (retail direct). Their shares are weighted by the proportion of fresh and processed food sold within the ABI/ABS retail classification.	CPIH Food (D7BU) and CPIH alcoholic drinks (D7CA). These are weighted by the split in sale of food and alcoholic drinks in ABS.
Food Wholesale	Geometric mean of retail and manufacturing sectors.	Geometric mean of retail and manufacturing sectors.
Non residential catering	GDP deflator	CPIH catering (D7CW) and CPIH alcoholic drinks (D7CA) weighted by their split within the sale of retail food and alcoholic drinks.
Wider economy	GDP deflator	GDP deflator

Food manufacture deflators

The Producer Price Index (PPI) is a monthly survey that measures the price changes of goods bought and sold by UK manufacturers.

The turnover deflator for food manufacture is used for the purchases deflator for food retail. To simplify the model the 'food products' series is used, a more inclusive estimate would be a weighted average of the three series:

• Food products K37L (G6SI from 2011)
• Soft drinks, mineral waters and other bottled waters JU5C
• Alcoholic beverages, including duty K38U

Food retail deflators

Retailing purchases deflator does not cover the divide between UK and foreign produced raw materials, which would have different price movements but as the detailed data on quantities purchased from other countries is not available it is not possible. By adjusting the turnover and output by the proportion of non-food items that are sold, it is possible to minimise the distortion caused by non-food sales.

Food wholesale deflators

A geometric mean is used. The geometric mean is the square root of the food manufacturing value multiplied by food retail.

Non residential catering deflators

CPIH catering and CPIH alcoholic drinks are used, weighted by their split within the sale of retail food and beverages. This assumes that the ratio of sales of food to alcoholic drinks is the same in catering as retail. Family Food data on purchases outside the home shows that this may not be the case with 19% of spending on eating out being on alcoholic drinks compared 13% of spending on household supplies in 2020/21.

Wider economy deflators

The measure used does not represent the full economy but rather non-public sector industries that are covered by the ABS. The main exclusions are financial sector and agriculture. GDP Deflator is the measure of domestically generated inflation. Further details are available here: https://www.gov.uk/government/collections/gdp-deflators-at-market-prices-and-money-gdp.

6. Quality changes

Quality improvements in products can be measured as increases in output. Deflating by price indices ensures that most quality changes are captured as volume changes. Changes in volume include quality changes, whilst changes in price exclude quality changes.

Change in current price value = (change in volume) x (change in price)

Retail productivity is possibly depressed by the omission of the non-market benefits offered by supermarkets, such as longer opening hours, and a greater product mix offered within stores. If more of the inputs are being spent improving the shopping environment then it will appear in the measure as inefficient use of these additional inputs and productivity will decrease.

7. Changes to methodology since 2006

Revisions to the methodology have been made mainly to take into account changes in the source data.

a) Revision to SIC2007 from SIC2003 for 2008 data onwards

The introduction of SIC2007 is the first major revision of the classification structure since 1992. The main difference between SIC2003 and SIC2007 that affects the productivity indicator is the reclassification of 'sale of automotive fuel' so that it is now classified as a retail activity. Previously, sales of fuel were considered part of the motor trade but now fuel sold through major supermarkets is included within the retail turnover data. They have been adjusted for as described in section 5. The producer price index uses SIC2007 for 2009 onwards

b) Change in the ABI (Annual Business Inquiry) to the Annual Business Survey (ABS) and Business Register of Employment Survey (BRES)

Previously average employment estimates were used, now point in time estimates are used throughout the series. UK 4 digit employment figures are not yet available for all industry sectors and drinks manufacturing estimate has been suppressed. Therefore some estimates are made to fill the gaps assuming that the rate of change year on year in employees all jobs series is the same as for the BRES series.

c) Change of method for rate of change of capital stocks

The flow of capital stocks had previously been estimated using a perpetual inventory method (PIM) within the model; this has been amended to use the rate of change in published capital consumption tables as explained in page 5 'capital input'.

d) Changes to deflators used

For the retail deflator the weighting of mostly raw compared to mostly processed has been changed to use APS retail sales data rather than the volume of sales in specialist shops: greengrocers, butchers and fishmongers.

The retail turnover and non-residential catering sector deflators were changed from RPI to the Consumer Price Index (CPIH).

Previously Agriculture Price Index (API) all items was used as part of the equation to deflate retail purchases (inputs). This has been amended to solely include four API elements: animals for slaughter and export, vegetables, potatoes, and fruit.

Previously all items RPI had been used for wider economy turnover deflator; this has been changed to GDP deflator.

From July 2015 the definitions of the sectors are the same as Defra's Food Statistics Pocketbook. These definitions are explained in the glossary. These changes mainly affect retail and wholesale.

From 2020 two of the producer price indices were changed. K37L and MC35 were discontinued by ONS and have been replaced with G6SI and GHHV respectively in this publication. These new PPIs have been backdated to 2011 (when they were introduced by ONS).

e) Changes to retail purchases and turnover

From 2008 onwards retail sales of motor fuel at supermarkets are included in the ABS data (arising from the change to SIC2007) representing a discontinuity in the series. They are adjusted for by removing the value of sales from turnover and proportionally the same amount from purchases in the retail sector.

f) Imputation

Missing data for Section C (Manufacturing), divisions 11 (manufacture of beverages), 46.17 (agents involved in the sale of food, beverages and tobacco), 46.3 (minus 46.35, i.e. wholesale of food, beverages and tobacco minus wholesale of tobacco products), 47.2 (minus 47.26, i.e. retail sale of food, beverages and tobacco in specialised store minus retail sale of tobacco products in specialised stores) and 47.81 (retail sale via stalls and markets of food, beverages and tobacco products) were imputed using the missForest “random forest” algorithm.

The non-parametric random forest approach has the advantage of being able to handle mixed types of missing data, address interactions and nonlinearity, scaling to high dimensions while avoiding overfitting and yielding measures of variable importance useful for variable selection (Tang 2017). Data is imputed by regressing each variable in turn against all other variables and then predicting missing data for the dependent variable using the fitted forest.

The method was implemented here by employing the missForest R package (version 1.5). Although the algorithm is able to consider data without a need for any allowance for categorical information, it was found in practice that overall data fitting was improved if variables were grouped together on a “like-for-like” basis, i.e.

Costs	Acquisitions
Total stocks and work in progress - value at end of year	Total employment - point in time
Total stocks and work in progress - value at beginning of year	Total employment - average during the year
Total stocks and work in progress - increase during year	Number of enterprises
Total turnover	Total capital expenditure - acquisitions
Approximate gross value added at basic prices	Total capital expenditure - disposals
Total purchases of goods, materials and services
Total employment costs
Total net capital expenditure

meaning that two separate sets of calculations were performed. Normalized root mean squared error (NRMSE) values for the Costs and Acquisitions groups were 5% and 13%, respectively.

8. Reliability

The quality of the calculation is dependent on the quality of the source data, in particular the Annual Business Survey. ONS publish some quality estimates for the ABS for certain industry sectors. Based on these estimates, it is possible to provide an indication of the reliability of the calculation (see external references for ONS website address).

In general, the trends revealed by the indicators are reliable but individual year on year changes are subject to sampling errors. The sampling error of a change in output between two consecutive years may exceed the change itself, especially when the change is small. For example, a change in output of 4% would have a standard error of the order of 15% to 25% of the change. There are similar errors associated with measurement of the inputs and with price deflation, although errors are not simply added together but balance out to some extent.

External references

Annual Business Survey and Annual Business Inquiry

Standard Industrial Classification (SIC) Archive

SIC 2007 Details

Capital Stocks, Capital Consumption and Non-Financial Balance Sheets

GDP deflator

Producer price index

Annual Survey of Hours and Earnings

Official guidance on legal minimum holiday entitlement

Stekhoven D. J., & Buehlmann, P. (2012). MissForest - non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), 112-118.

Daniel J. Stekhoven (2013). missForest: Nonparametric Missing Value Imputation using Random Forest. R package version 1.5.

Tang F., & Ishwaran, H. Random Forest Missing Data Algorithms. Stat Anal Data Min. 2017 10(6), 363–377.

Glossary of economic terms

Current prices

Prices not expressed in terms of the base year i.e. in today’s terms.

Constant prices

A series expressing value in terms of the prices of a single year, removing the effects of inflation.

Real terms

Figures that have been adjusted for changes in the purchasing power of money.

Contact details for enquiries

David Lee

Tel: +44 (0) 208 026 3006

Email: david.lee@defra.gov.uk

Defra,
2nd Floor, Seacole Block,
2 Marsham Street,
London
SW1 4DF