Research and analysis

MEGS evaluation: full report on a rapid response evaluation of the DIT Music Export Growth Scheme (MEGS) (executive summary)

Published 4 July 2023

Read the MEGS evaluation: full report on a rapid response evaluation of the DIT Music Export Growth Scheme (MEGS) in PDF format.

This is a report on the results and conclusions from a rapid survey-based evaluation of the Music Export Growth Scheme (MEGS). The aim of the project was to provide the Department for International Trade (DIT) with an improved evidence base for assessing the benefits generated by MEGS.

The study involved providing DIT with a presentation slide-based set of Interim Findings over a very tight timeline in order to support a business case submission to HM Treasury for MEGS extension funding for one year. These interim results were submitted as required on the 16 April 2021. Subsequently, and in line with an approach outlined in the Interim Findings, attention in the study then switched to providing DIT with a full report with an ‘augmented’ emphasis on:

• providing more robust econometrics-based results on the economic impact of MEGS, and on;
• articulating a Theory of Change, Logic Model and other aspects of potential use to DIT in the forthcoming Comprehensive Spending Review (CSR). Consequently, the main focus of this report is on taking forward and further developing the CSR-oriented aspects of the study rather than simply on providing a document repeating the findings already provided in the slide-based Interim Findings that have already been provided

The study had a strong ‘proof-of-principle’ dimension in the sense that it has sought to demonstrate what it will be possible to achieve in the future as regards effective monitoring and evaluation (M&E) of MEGS (or any future scheme with this focus). This aspect its important because prior DIT experience with a survey-based evaluation of MEGS had not generated credible results. However, this study has demonstrated that, handled appropriately, a survey-based evaluation can generate useful and credible findings. Consequently, some specific recommendations are made as to how MEGS M&E arrangements could be organised moving forward in a way that ‘mainstreams’ evaluation into the application and follow-up process.

The main findings and conclusions in this report are as follows:

• MEGS support for UK music industry exports operates, in effect, via an investment portfolio ethos – public funding plays a catalytic role in enabling longer-term (5 year plus) gains in markets where export facilitation cost barriers would otherwise limit export potential (notably in the USA)
• this reflects the familiar risk-reward relationship: the greatest rewards in export performance tend to entail taking the greatest financial risks - hence limiting the export performance of emerging acts lacking the necessary financial resources
• the MEGS grant allocation process favours acts with an established export track record, who are consequently well-placed to build-on their experience by penetrating new overseas markets in which there are significant cost hurdles and a threshold level of business experience and acumen is likely to a key success factor
• declined MEGS applicants have a lower initial export profile and, as a result, their growth in exports takes place from a lower base, which complicates estimating the exports for MEGS funded acts likely to have happened anyway
• this major difference between declined and successful MEGS applicants means that the evaluation must consider the implications of the fact that it is far easier to increase exports from a low starting point than from a higher export level. In turn, this creates a risk that the evaluation will over-state the increase in exports for successful MEGS grants likely to have happened anyway – thus under-estimating MEGS additionality
• MEGS funding initiates export boosting activities in the near-term that are viewed as tipping the longer-term odds of export success in favourable ways
• an econometric analysis incorporating modelling of the likely longer-term impacts up to 5 years was carried out. Given sample size and data limitations this was only intended to generate indicative results for subsequent validation via follow-up M&E work. This reflects the ‘proof-of-principle’ aspect of this study – aimed at demonstrating what it will be possible to achieve in the future. This analysis required the following assumptions to be made: (a) the treatment and comparator group were similar to one another at the point at which the decision to provide support was made, (b) the treatment and comparator group were “trending together” prior to the intervention, (c) there are no other factors (outside the programme support) which may impact upon the 2 groups differently, (d) The distribution of the level of exports in the 2 groups is normal. To aid this assumption, the data for exports has been transformed using a natural logarithm function, which has transformed the export values of the 2 groups into a normal distribution
• the major, but unavoidable, limitation of this indicative econometric analysis is that it required assumptions to be made that ignored the strong differences between the declined and successful MEGS applicants noted above (successful applicants are already stronger export performers). In future evaluation analyses using a larger sample it will be advisable to use Propensity Score Matching (PSM) to handle this aspect – which will generate more accurate econometric results
• the results indicate a positive, significant difference-in-difference impact of the programme, with coefficients of between 1.5 and 3.2. This would indicate that the programme has an effect of increasing exports among participants of between 158 percent (over 2 years) and 320 percent (over 5 years). However, these effects should be viewed with caution due to technical caveats, and the relative low explanatory power of the models (low R2) values due to a limited sample size for this proof-of-principle exercise
• this demonstration that a difference-in-difference assessment of the impact of the programme is technically feasible (even though this pilot application faced sample size and consequent analytical constraints) means that estimating the benefit cost ratio (BCR) for MEGS is possible. The major challenge in BCR estimation is that MEGS support attracts relatively strong commercial investment at a rate of over £2 for every £1 of MEGS funding (estimates derived from a useful analysis carried out by the BPI subsequent to the survey work discussed here). This leverage demonstrates that the public funding is being significantly augmented by private investment and reflects the strong commercial ethos in MEGS funding allocation (in the sense that MEGS helps to tip the odds to threshold levels that then leverages additional private sector investment). This positive signal as regards MEGS means that the additional private sector investments must be factored into the BCR estimate because the 3 to 5 year difference-in-difference forecast produces unrealistic BCRs if the additional private sector investment is not considered
• when leveraged private sector investment is factored-in, and with reasonable ‘rule of thumb’ assumptions about MEGS administration costs the BCR at year 1 is 1.28, 1.8 at year 2 and 4.86 at year 3. Subsequent BCRs are higher – but are less credible given the limitations of this ‘proof of principle study’ depending on a small sample size that limited the ability to factor-in the different export propensities of funded and declined MEGS applicants
• the long-term return-on-investment in exports generated due to MEGS funding are most accurately assessed at a portfolio-level where a skewed distribution is the be expected
• the international music market is not a level playing-field – subsidies and overseas government support can play a key role in influencing the odds of success and failure in export growth. Hence, MEGS helps to offset this competitive dis-advantage for the UK music industry. Consequently, the ‘denial cost’ of ceasing MEGS support would be reflected in increased risk exposure to UK music exports that will only become apparent several years into the future
• the significant lag between grants and eventual benefits means that by the time the consequences of ending MEGS funding are apparent it will, for practical purposes, be too late to intervene to reduce the damage to music exports caused. This is an important issue because music streaming has changed how music is made and distributed and is opening markets across the world. Consequently, speed to market is critically important if the UK music sector is to maintain and grow its global market share of recorded music
• this lagged benefits pathway means that a new ‘fit-for-purpose’ monitoring and evaluation framework would need to be put in place if MEGS continues over the longer-term. This framework would require annual updates on overall industry export performance, identifying any changes in market access barriers that may require DIT intervention. If implemented via an online reporting system this portal would allow for a comparison between funded and declined MEGS (and potentially ISF grants) over the long-term. By requiring export tracking portal registration for all MEGS and ISF grant applications, this system would be able to compare long-term export performance for funded and declined MEGS and ISF applicants (and indeed any other DIT export support). Such a system would eliminate the reliance on forward estimates for 3 and 5 year impacts used in the pilot econometric analysis in this study
• responses to this survey and associated liaison indicates that the music industry may support an export tracking and trade barrier identification system of this type – as concerns over the future are high