Research and analysis

LMI for All 2017 to 2018

Updated 25 February 2021

© Crown copyright 2021

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/developing-and-enhancing-a-labour-market-information-database-lmi-for-all/lmi-for-all-2017-to-2018

Overview

What is LMI for All?

It was created as an open data project, which supports the wider government agenda to encourage the use and re-use of government data.

The LMI for All service covers 3 strands:

  • technical infrastructure
  • data development
  • stakeholder engagement

It aims to:

  • identify and investigate which reliable sources of LMI can be used to inform the decisions people make about learning and work
  • bring these sources together in an automated, single, accessible place, referred to as the LMI for All database
  • support the use of the database by developers to create career websites and apps to help the users make better decisions about their learning and work
  • promote the LMI for All service

The importance of LMI in careers

High quality, reliable and robust LMI is vital to offering effective career practice as it helps:

  • navigate complex labour markets
  • identify a pathway to goals and ambitions
  • inform career decision making
  • our understanding of a changing labour market

What data is in LMI for All?

The LMI for All database includes:

  • employment: historical, current, projected and replacement demand (Working Futures based on Labour Force Survey (LFS) and the Business Register and Employment Survey (BRES))
  • pay and earnings (estimates based on Annual Survey of Hours and Earnings (ASHE) and LFS)
  • hours (ASHE)
  • unemployment rates (LFS)
  • number of vacancies (Employer Skills Survey (ESS))
  • higher education destinations (HESA)
  • occupational descriptions (Office for National Statistics (ONS))
  • skills, knowledge, abilities and interests (O*NET)
  • current vacancies (FindaJob)

How can the data be used?

There are 3 ways in which to access and use data in the LMI for All database. You can:

  • develop a web interface or mobile application that uses the API to access the LMI for All database
  • embed a Careerometer, a free widget, into your own website
  • explore a third party website or application that uses LMI for All data

Careerometer widget

LMI for All provides an easily customisable widget for websites which:

  • organises data by occupation: the user simply types in the title of the job they are interested in and the widget provides a series of options from which the user can select the most relevant
  • provides access to a selection of headline data relating to pay, weekly hours of work and future employment prospects
  • includes the core elements of a role: occupational description, earnings, hours and employment prospects
  • offers comparisons between occupations facilitated
  • targets users: learners, parents and carers, teachers and career professionals

Companies using LMI for All

  • Eluceo
  • Adviza
  • Prospects
  • icould
  • SkillsPlanner
  • Active
  • Department for Work & Pensions
  • Skills Development Scotland
  • SACU
  • Cascaid
  • RCU
  • KareerHub
  • Skills Match London
  • Education & Skills Funding Agency
  • U-Explore
  • Next Steps
  • Pathways
  • Moving on

Questions about the labour market

Title Questions Data sets in LMI for All
Employment levels by occupation How many jobs are there? How many in my area? What are the past trends? What are likely future trends? Labour Force Survey (LFS), Working Futures
Average earnings by occupation How much do people get paid for this job? How much at the start of their career? How much in my area? Annual Survey of Hours and Earnings (ASHE)
Unemployment by occupation What proportion of people in this occupation are currently out of work? Annual Population Survey (APS)
Profile of qualification level by occupation What level of qualification do people have in this jobs and what am I likely to need? Labour Force Survey (LFS), Working Futures
Vacancies by occupation How many vacancies are there for this job? What proportion are hard to fill? Employer Skills Survey (ESS)

What the LMI team do

The LMI for All team:

  • update and maintain the LMI for All database
  • explore sources of open data for inclusion in the database
  • promote the LMI for All service and websites and applications using LMI for All and Careerometer
  • work with developers who are producing web interfaces and applications using LMI for All
  • work with a range of stakeholders to provide advice and guidance on designing web interfaces and applications using LMI for All
  • examine the implications for careers practice of this resource

Technical infrastructure

The LMI for All service technical infrastructure details the Application Programming Interface (API) and the API Explorer. If you’re interested in using LMI for All there are suggested pathways through the data and details of the support offered to developers.

The aims of the technical infrastructure strand are to:

  • maintain a secure and robust infrastructure for the data providing an environment that can be extended in the future as new data becomes available and as the number of end-users increases
  • implement processes to ensure that the data made available to developers and end-users meets required quality standards
  • provide modern and flexible software tools to allow the querying of the database by external users
  • provide software tools and spaces for documenting the process and allowing public access to the LMI for All database

Technical infrastructure

  • cloud based secure servers for accessing linked data
  • systems and procedures for data quality control
  • Application Programming Interface (API) for developers to query database
  • extract, transform and load (ETL) systems to semi automate data uploads - to support the refreshment of existing datasets and integration of new datasets and to maintain the quality of the data
  • data cube - a set of data that is usually constructed from a subset of a data warehouse and is organised and summarised into a multidimensional structure defined by a set of dimensions and measures

Application programming interface (API)

  • using the LMI for All API queries across one or more of these tables can be performed, which may either be joined or in sequence, according to the information requested
  • the information is then collected and translated into a universal data format (JSON), before being sent back to the client
  • JavaScript Object Notation (JSON) is a lightweight data structuring standard
  • Application Programming Interface (API) is a set of routines, protocols and tools for building software applications

The LMI for All API:

  • makes it easier to develop a program by providing all the building blocks, which a programmer then puts together
  • allows basic queries as it is reachable via HTTP and speaks JSON
  • enables more complex queries to be formulated as a JSON structure

The team uses a dashboard to help them monitor the database and the API. It checks the overview of the API ‘health’.

Design a web interface or an application using LMI for All

If you want to develop a web interface or an application using the LMI for All database:

  • have a look at some of our third party users
  • review the accounts of those who have developed an interface to LMI for All
  • take some inspiration from the Career Hack competition winners
  • think about the different pathways through the data and what users may find useful when thinking about future work and learning pathways

Some examples of how others have combined data from LMI for All with other data (such as careers videos and local data) include:

JobHappy

Pathways through data

Users can:

  • get a list of occupations by selecting their skills and interests
  • search a list of occupations

This filters results to show:

  • job description
  • workforce profile
  • employment statistics
  • pay
  • vacancies

Support for developers

You can:

Data development

The aims of the data strand are to:

  • refresh existing data sets held on LMI for All
  • manage documentation relating to the datasets
  • check to ensure data rendered by the Application Programming Interface (API) are consistent with values contained in the dataset generated by the data development team
  • advise on issues and questions relating to the data as part of customer care support arrangements
  • contribute to internal data management processes, including liaison with the technical team around documentation and quality assurance of data

Overview of data sources

Data for the LMI for All database are drawn from the following data sources and owners:

Organising framework for LMI for All

Data is linked together by occupation and classified at the unit group 4-digit level of the 2010 Standard Occupational Classification (ONS) (SOC).

An index of around 28,000 job titles mapped to SOC enables the end-user to search for and access data of interest and relevance to careers in an intuitive fashion.

Data we use

Occupational descriptions

  • source: Standard Occupational Classification (ONS), Office for National Statistics
  • based on UK Standard Occupation Classification (SOC) index at 4-digit level
  • provides description of occupational role and common tasks associated with the role
  • provision - UK SOC2010 4-digit
  • minor updates when required

Questions you can ask of the data:

  • What is this occupation about?
  • What are the common tasks associated with a particular occupation?

Skills, abilities, knowledge and interests

  • source: US O*NET
  • mapped to UK 4-digit occupations
  • provides attributes and descriptors for each role, including:
    • skills - developed capacities that help learning and knowledge acquisition (for example, basic, problem solving, resource management, technical)
    • abilities - attributes that influence performance (for example, cognitive, physical, sensory)
    • knowledge - organised sets of principles and facts (for example, clerical, design, education, English language)
    • interests - preferences for work environments and outcomes (for example, realistic, social, investigative)
  • provision - UK SOC2010 4-digit
  • updated annually

Questions you can ask of the data:

  • What skills do I need for this occupation?
  • What interests are associated with a particular occupation?

Employment (current, past, future) and replacement demand

Questions you can ask of the data:

  • How many people are employed in a particular occupation?
  • What is the future demand for an occupation?

Unemployment rates

Questions you can ask of the data:

  • How many people are unemployed in a particular occupation?
  • What are the unemployment rates for an occupation over the last 5 years?

Pay and earnings, weekly hours

  • source: produced by the Institute for Employment Research (IER)
  • provision - detailed estimates based on econometric analysis of LFS and Annual Survey of Hours and Earnings (ONS) data, constrained to match ONS published headline figures
  • updated annually

Questions you can ask of the data:

  • What is the weekly pay in a particular occupation?
  • How much does a 30 year old earn in a particular occupation?
  • How many hours do people work in a particular occupation?

Graduate destinations

  • source: HESA
  • provision - analysis of higher education courses previously undertaken by people in particular jobs, data from HESA destination of leavers survey
  • updated annually

Questions you can ask of the data:

  • What occupations did people gain after a particular higher education course?
  • What level of higher education course was undertaken by those employed in a particular occupation?

Vacancies (incidence and causes)

  • source: Employer Skills Survey
  • provision - number of vacancies, hard-to-fill vacancies, skills shortage vacancies for 2013, 2015, 2017
  • updates approximately every 2 years

Questions you can ask of the data:

  • How many vacancies are reported in a particular occupation?
  • What percentage of all vacancies are hard-to-fill?

Geographic patterns of employment

  • source: Census of Population (ONS) (England and Wales)
  • provision - 2011 only, information on geographical patterns of employment and travel to work distances
  • updated every 10 years

Questions you can ask of the data:

  • How far do people travel to work in a particular occupation?
  • Where do people in a particular occupation live?

Live vacancies

  • source: Department for Work and Pension
  • provision - live vacancies from find a job
  • updated continuously

Populating the database

  • tapping into existing official and other sources of data
  • focus on detailed occupations over 350 4 digit categories using SOC2010
  • most data sources are not designed for that purpose, therefore estimates are used based on econometric and other methods for the key building blocks, for example:
    • employment and replacement demand - produced using the Working Futures database which combines Labour Force Survey and Business Register and Employment Survey data as part of the econometric model
    • pay - produced using a combination of data from Labour Force Survey and Annual Survey of Hours and Earnings

Quality assurance and data integrity

  • data released through the LMI for All service meets the conditions on which data providers make data available
  • data do not breach confidentiality nor are disclosive
  • data which might potentially disclose information about respondents have been checked and transformed into a synthetic form or into information from a statistical model
  • some datasets have potentially disclosive data suppressed, for example:
    • sample sizes of less than 50 from ESS are suppressed
  • statistical modeling has been used for employment data from the Working Futures dataset
  • estimates of pay at 4 digit SOC are produced through a RAS model

Data refreshment cycle

Every year Every 2 to 3 years Other
Pay and earnings Employment (historical and current) Every 10 years - geographical patterns of employment and travel to work distances
Hours Employment (projected) Continuous - current vacancies
Un/employment rates Employment (replacement demand)  
Destination of graduates Number of vacancies  
Occupational structure and description Number of hard-to-fill and skills shortage vacancies  
Occupational skills, knowledge, abilities and interests    

Stakeholder engagement

The aims of the stakeholder and engagement work were to:

  • encourage third parties to exploit this free resource by developing applications or enhancing existing applications
  • increase the reach of LMI for All among end-users
  • encourage third-parties to implement the LMI for All Careerometer
  • drive end-user traffic to existing applications that offer access to LMI for All data
  • foster positive and supportive attitudes towards the LMI for All service
  • consider implications for careers practice

Engaging stakeholders

During 2017 to 2018, we undertook a range of activities to engage with a range of stakeholders. This work included:

  • disseminating the LMI for All service, current users, case studies and Careerometer
  • webinars
  • online support
  • reviewing and updating the LMI for All website and case studies
  • delivering LMI for All training at CDI professional development events

Case study: Next Steps Cornwall Career Card Challenge

The Next Steps Cornwall Career Card Challenge:

  • provides valuable LMI data
  • supports teachers and advisers with an engaging activity
  • increases knowledge around pay and % workforce in 100 selected careers/ job roles in a fun and informative way
  • has developed into the Next Steps Challenge App

Inspiring stakeholders

Through a range of activities, we have been inspiring others to use the LMI for All service by:

  • supporting those wishing to develop an interface
  • promoting those third party sites that use LMI for All
  • encouraging others to embed the LMI for All widget, Careerometer, into their own website
  • considering practice implications

Updates from our users: 2017 to 2018

  • Eluceo have more users signed up and they are developing their app
  • Start by U-Explore have significantly increased their users and they have expanded the data they use from LMI for All
  • icould have added more information to the career dashboards
  • Pathways has launched its widget which uses LMI for All data combined by course and HESA data
  • Next Steps Cornwall have launched their challenge app based on their card game
  • Kareerhub launched their website and have users signed up

Current statistics on usage

The current statistics on LMI for All usage shows that:

  • usage is continuing to increase - doubling over the last year
  • the best base-line minimum is estimated at 60,000 users
  • daily queries to the API are on average 600,000 per day (peak 800,000 per day)
  • the LMI for All service manages around 4.7 queries per second
  • Working Futures and the Annual Survey of Hours and Earnings remain the most popular datasets accessed through the API
  • Careerometer is estimated to be deployed in over 65 websites (likely to be higher as it can be embedded in restricted access sites)

API queries (01/08/18 to 30/09/18)

Daily queries to the API are on average 600,000 per day (peak 1.047 million per day). This is an increase from around 400,000 queries per day in the same period a year ago.

Usage of the API is lower over the summer period.

Datasets accessed

Dataset % accessed
ASHE 26%
ESS 7%
O*Net 4%
O*Net 2%
SOC 17%
Vacancies 2%
Working Futures 41%

Source: Working Futures (employment data), Annual Survey of Hours and Earnings (ONS) (pay and hours) and Standard Occupational Classification (ONS) (occupational descriptions) are the most popular datasets accessed.

What’s next for LMI for All?

Upcoming activities for 2018 to 2019 include:

  • a data refresh ensuring users can access the most up-to-date information
  • new occupational mappings for apprenticeship standards enabling access to current apprenticeship vacancies
  • development work for occupational changes to ensure data are current
  • reviewing and maintaining technical infrastructure to ensure a fast and reliable service as user numbers increase
  • exploring the role of AI to match courses and occupations providing a new pathway through the database
  • designing and developing a new widget with a regional element
  • redesigning and updating the LMI for All website to help users access the service
  • undertaking a range of stakeholder engagement activities
  • supporting third party users evaluate their web interface or app to ensure we are meeting user needs

Policy background

The open data white paper talks about using open data more effectively.

LMI for All was created as an open data project, which supports the wider government agenda to encourage the use and re-use of government data. We aim to:

  • improve access to publicly funded datasets
  • improve quality of LMI for mobility and transitions
  • enhance careers services

We can do this by:

  • accessing a number of publicly funded and large-scale, long-standing databases which have been limited despite the importance of LMI for supporting career decisions
  • strengthening and improving the quality of LMI for careers and employment practice to help individuals who wish to start or restart working, or wish to move between jobs
  • providing easy access to improved data in one place to enhance career services

The careers strategy also sets out the government’s plan to improve social mobility and offer opportunity to everyone.

Contact us

LMI for All

http://www.lmiforall.org

Labour market information (LMI) for All.

Back to top