FCDO’s review of the Independent Commission for Aid Impact

Question 1

Foreword by the Foreign Secretary

Accepted Answer

The Independent Commission for Aid Impact (ICAI) plays a vital role, providing independent evaluation and scrutiny of the UK’s aid spending. With this review, our aim is to make sure that ICAI is focused on performing that role to the highest possible standard, by providing practical recommendations to support the Government to deliver the greatest impact from UK aid.

We have extensive experience and expertise in the Foreign, Commonwealth and Development Office (FCDO) but an independent voice to provide additional challenge is invaluable. For almost ten years ICAI has been that voice. This review of ICAI is about valuing and strengthening ICAI to make it even more effective and action-oriented. As we continue to build our capabilities in the FCDO and play a greater coordinating role across Whitehall, we need ICAI to play a strong part in ensuring that we know what works, and where we can improve.

The review tells us that ICAI can play this role and can drive better results. A high-performing ICAI will continue to drive even better performance at the FCDO.

ICAI can maximise its impact by focusing on practical steps, designed to help Government to learn in a productive and constructive way. This is a key point. Whilst independent of Government, ICAI and the FCDO need to work together to drive learning and optimise impact from their work.

Many of the recommendations contained in this report are for ICAI and my teams at the FCDO (and across Government) to take together. ICAI and the FCDO can drive better decision-making and more effective aid, together achieving even greater impact for some of the poorest and most vulnerable people around the world.

Question 2

Executive Summary

Accepted Answer

The Independent Commission for Aid Impact (ICAI) provides strong external scrutiny of UK Official Development Assistance (ODA) and offers excellent support to Parliament in its role in holding the Government to account. It has served as a model for other donor countries in how to ensure transparency and value for money in aid spending. ICAI should continue to fulfil its role for ODA spent by both the new Foreign, Commonwealth and Development Office (FCDO) and other Government departments.

The driving force behind scrutiny is to improve how things are done. Recommendations that improve the impact of UK aid should be at the centre of all that ICAI does and should be the spirit in which Government, and others, work with ICAI. This review has found that ICAI can do more to ensure it helps Government deliver the best possible impact for UK aid, and for its reviews to give practical action-oriented recommendations.

In Chapter One, the review asked: what is ICAI for?

Recommendations 1 to 4 address ICAI’s remit and overarching objectives.

Recommendation 1: Focus ICAI’s overarching remit to better support Government learning through an increased emphasis on practical recommendations for future aid work, in addition to its current focus on independent evaluation and scrutiny.

Recommendation 2: FCDO to appoint a lead senior official for ICAI, who should have quarterly dialogues with the Commissioners.

Recommendation 3: ICAI should plan its work with the Government’s strategic objectives for ODA in mind, which Government should share with them. This will enable ICAI to assess Government delivery against those objectives.

Recommendation 4: ICAI should specify in the Terms of Reference / scope of each review how it complements and does not duplicate other external scrutiny processes.

In Chapter Two, the review asked: how does ICAI carry out its work?

Recommendations 5 to 11 address ICAI’s workplan, its reports, recommendations and how Government responds, as well as opportunities for wider learning.

Recommendation 5: ICAI Commissioners will continue to exercise full control and make final decisions over ICAI’s workplan. The quarterly dialogues with the new ICAI lead senior official should be used to have regular discussions about the workplan with FCDO.

Recommendation 6: To help build up a body of best practice in establishing and managing aid programmes, where there are lessons that have wider implications beyond the subject in focus, ICAI should include specific learning points in its reviews.

Recommendation 7: ICAI should consider developing, in consultation with the Government, partners and the IDC an improved approach to assessment ratings in reviews, by 2023.

Recommendation 8: ICAI should consider the recommendations from the World Bank report and continue to reflect best practice in their own recommendations.

Recommendation 9: FCDO should determine the best internal committees and reporting structures to monitor recommendation uptake, ensure Government responses are completed on time, and consider systemic issues and wider learning points arising from recommendation 6 – and make that process clear to ICAI.

Recommendation 10: The FCDO Management Board will meet annually with ICAI Commissioners, with the option of including Directors-General from key ODA-spending departments as needed. This meeting should focus on learning and reflection on where ICAI reviews are having most impact now and could have most impact in the future.

Recommendation 11: ICAI and FCDO to consider together how to expand ICAI’s toolkit beyond reviews and reports to include more informal ways of helping organisational learning (e.g. workshops, roundtables, webinars), and how ICAI’s reviews and annual report can capture best practice.

In Chapter Three, the review asked: how well does ICAI carry out its work?

Recommendations 12 to 15 address ICAI’s efficiency and effectiveness, performance management, and value for money.

Recommendation 12: ICAI should continue to develop and adapt its KPIs alongside refocusing its remit in order to give effect to recommendations in this report by March 2021.

Recommendation 13: Going forward, new Commissioners should continue to be appointed for a single term, but their appointments should be staggered. Timing could also be better sequenced with supplier procurement.

Recommendation 14: To ensure it can review effectively an expanding range of ODA instruments and programmes, ICAI should ensure it has the right skills and expertise, both in house and through its supplier.

Recommendation 15: FCDO to investigate options for secondments of Government staff to the ICAI Secretariat and review teams, and vice versa, putting sufficient conflict of interest protocols in place.

Question 3

Introduction

Accepted Answer

The Independent Commission for Aid Impact (ICAI) was established in May 2011 to help scrutinise and improve the quality of UK aid spending. Almost ten years on, and alongside the launch of the Foreign, Commonwealth & Development Office (FCDO), the Foreign Secretary announced that he wanted to strengthen the role of ICAI and commissioned this review.

The review’s objective was to ensure that ICAI’s remit and operating model are fit for purpose for improving the impact of UK aid and identifying what works in line with the Government’s aims for Official Development Assistance (ODA).

The review did not assess ICAI’s status as a Non-Departmental Public Body. The location of ICAI’s office (currently based in Whitehall) was not in the scope of this review, however, the Government’s ongoing Places for Growth Programme will revisit this.

This review builds on the findings of the 2017 Tailored Review of ICAI, which concluded that ICAI’s functions were still required, subject to some refinements to promote clarity and maximise value for money.

A small review team of FCDO officials was appointed to gather evidence, analyse the results and write this report. The team reported to an internal steering committee, led by an FCDO Non-Executive Director. The team also convened a Challenge Panel of independent experts.

To inform the review, Ministers and officials conducted interviews with ICAI’s Commissioners and secretariat, the International Development Committee (IDC), the Sub-Committee on the Work of ICAI, ^{[footnote 1]} several civil society organisations, other scrutiny bodies, and Government departments that had been reviewed by ICAI. The team also received a range of written contributions, including from members of the public.

The report is divided into three sections:

What ICAI is for (remit and function)
How ICAI does it (approach and methods)
How well ICAI does it (efficiency and effectiveness)

Question 4

Chapter one: what ICAI is for

Accepted Answer

ICAI’s remit

1) ICAI was established in May 2011 by the then Secretary of State for International Development, Andrew Mitchell, to “shine a light on where aid works and where improvements are needed.” ^{[footnote 2]} The creation of ICAI was world-leading and its contribution to improving the impact of UK aid cannot be disputed – its scrutiny improves how aid is delivered, providing assurance to UK taxpayers and improving the lives of the world’s poorest.

2) ICAI must be independent of all, presenting robust analysis, challenging assessments and ambitious yet practical recommendations derived from evidence on best practice. But these findings and recommendations must lead to change, because the driving force behind scrutiny is to improve how things are done. This should be at the centre of all that ICAI does, and should also be the spirit in which Government, and others, work with ICAI. This role and emphasis on helping Government to learn more and to learn better from ICAI, both from what works and where we must all do better, should be put front and centre in ICAI’s remit. All of the recommendations in this report seek to deliver this overall ambition and to drive improvements in the way UK ODA delivers impact.

Recommendation 1: Focus ICAI’s overarching remit to better support Government learning through an increased emphasis on practical recommendations for future aid work, in addition to its current focus on independent evaluation and scrutiny.

3) The learning from ICAI’s reviews needs to land with those who have the power to enable change, so ICAI needs to have a strong partner and advocate in Government. ICAI works best when it is a true critical friend to Ministers and the department, supporting the IDC’s public scrutiny, but also with the confidence to have a direct line. Current and former Ministers place high value on the opportunity to hear difficult messages, and on hearing them from a voice that is independent of the department they are running. ICAI needs to be a critical friend to help departments to deliver aid more effectively, and they should seek to engage Ministers as they feel appropriate.

Recommendation 2: FCDO to appoint a lead senior official for ICAI, who should have quarterly dialogues with the Commissioners.

4) The Foreign Secretary’s 26 November statement to the House of Commons ^{[footnote 3]} sets out the Government’s strategic objectives for ODA in light of the 2020 Spending Review. The Spending Review also gave the Foreign Secretary a greater role in coordinating and ensuring coherence of development spending across Government. Building on the opportunities provided by the merger of the former Foreign & Commonwealth Office (FCO) and Department for International Development (DFID), this ‘double lock’ will drive greater strategic use of ODA. Formalising the role of a lead senior official for ICAI should therefore be able to give ICAI Commissioners greater insight into the whole of UK ODA, better supporting their work.

Recommendation 3: ICAI should plan its work with the Government’s strategic objectives for ODA in mind, which Government should share with them. This will enable ICAI to assess Government delivery against those objectives.

Relationship with other aid scrutiny

5) The review has identified four forms of internal scrutiny (from individual programme reviews to internal audit) and five forms of external scrutiny (from Parliament to OECD/DAC peer reviews) on the aid budget (see Box 1). It is right that UK taxpayers’ money spent overseas receives thorough scrutiny, and that FCDO takes its own internal scrutiny seriously. The FCDO could usefully share more information with ICAI about planned internal scrutiny processes.

6) ICAI’s independent and external scrutiny role is critical. However, in such a crowded field with each body having different angles and remits, there is scope for enhanced collaboration, both in timing and coherence, to get the clearest picture of how UK aid is being delivered. It is in no one’s interest to duplicate work. ICAI coordinates well with the National Audit Office (NAO), which is the UK’s independent public spending watchdog, but this approach needs to extend to the other scrutiny forms too. ICAI and the Government together could do more to help ICAI deconflict its workplan with other aid scrutiny.

Recommendation 4: ICAI should specify in the Terms of Reference / scope of each review how it complements and does not duplicate other external scrutiny processes.

Box 1: scrutiny of the aid budget

External Scrutiny

1) Independent Commission for Aid Impact (ICAI) undertakes circa eight reviews per annum scrutinising UK aid spending.

2) International Development Committee (IDC) scrutinises the Official Development Assistance (ODA)-related spending, administration and policies of FCDO and monitors the expenditure of ODA by other UK Government departments. It is supported by the Sub-Committee on the Work of ICAI, which scrutinises ICAI’s reports on value-for-money of ODA expenditure and assesses the FCDO’s and other Government Departments’ responses to those reviews.

3) National Audit Office (NAO) scrutinises public spending for Parliament through value for money studies of FCDO’s policies and programmes.

4) Public Accounts Committee (PAC): scrutinises the value for money of public spending.

5) Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Peer Review: The OECD DAC facilitates Peer Reviews of Member Countries on a five-yearly basis. Reviews assess whole-of-Government development policy, programmes and systems, as part of a DAC-wide commitment to continuous learning.

Internal Assessment and Evaluation in FCDO

6) Internal Audit Department (IAD) provides independent, risk-based assurance across FCDO. It also investigates allegations of fraud, corruption or safeguarding.

7) Quality Assurance Unit (QAU) provides internal, independent quality assurance of major business cases. Its reviews are based on evidence, combine inputs from a range of specialists, and provide an overall assessment of value for money and recommendations.

8) Annual Reviews (ARs) and Project Completion Reviews (PCRs) are undertaken on all ex-DFID programmes including assessment of progress towards impact and outcomes. Annual reviews are published to Devtracker. The Portfolio Quality Index (PQI) is useful in assessing the performance of a portfolio of programmes. It is a weighted average of scored ARs and PCRs in the prior twelve months. Annual reviews are also undertaken on all ex-FCO programmes.

9) Monitoring, Evaluation and Learning (MEL): The Evaluation Unit provides support to FCDO (and others) on monitoring and evaluating ODA; strengthening standards, quality and design of evaluation and generating robust evidence to support DFID policy priorities and fill critical evidence gaps. Targeted MEL functions also exist to provide similar support to ex-FCO programmes.

Note: the information in this box is a combination of former DFID and FCO practices and is correct at the time of publication.

Question 5

Chapter two: how ICAI does it

Accepted Answer

This section considers ICAI’s workplan, its reports, recommendations and how Government responds, as well as opportunities for wider learning.

ICAI’s workplan

7) ICAI aims to have a rolling 18-month workplan, reflecting the substantial work involved in conducting reviews, and this is published online. ICAI’s website sets out the criteria it uses to develop the workplan, including consultation with stakeholders, and civil society report that they find this approach helpful. The workplan is then discussed and agreed with the IDC. However, there is currently no formal consultation with Ministers or senior officials.

8) ICAI can remain fully independent while seeking input on its workplan from a wide range of stakeholders. That process of consultation and consideration ought to include all relevant parties on the same footing. ICAI must of course retain full control of its workplan, which it agrees with the IDC, but in the spirit of ensuring maximum learning and impact from ICAI’s work, it is essential that ICAI look to involve departments from the outset. This engagement could help identify further areas where scrutiny and challenge are needed to drive learning and improvement. This will help ensure that the FCDO and others understand the driving force behind review topics and can make the best use of ICAI’s advice and recommendations.

ICAI’s reviews

9) ICAI reviews have expanded from their original single-model set up to conducting three types of review: full thematic reviews (including country portfolio reviews and results reviews), rapid reviews, and information notes.

10) Some information notes were produced during earlier ICAI phases, but they have become more common since 2019. These can be commissioned by the IDC as shorter, factual papers that provide information and analysis on an area of UK aid, but do not reach evaluative judgements or provide recommendations. There is no doubt that these notes are of value, particularly to the IDC. However, other bodies can fulfil this function, for example committee clerks or the Library of the House, and in deciding how to spend limited resources, ICAI should monitor carefully the cost of information notes to ensure sufficient capacity for both full and rapid reviews.

11) The quality and credibility of reviews is generally considered to be high. There were some concerns raised about ICAI’s flexibility to react to changing situations (in light of the lengthy timescales of full reviews), and the need to ensure shared understanding of the scope of the review and the sorts of insights that might be gained at the outset. Reports tend to be backward-looking and focus on qualitative discursive analysis. Although reports provide a thorough overview of what has happened in an area and the impact it has had, their findings do not always currently translate into learning that can be applied to a wider set of issues. The ever-evolving development landscape means that reports have less use in, for example, developing high quality business cases for future work.

Recommendation 6: To help build up a body of best practice in establishing and managing aid programmes, where there are lessons that have wider implications beyond the subject in focus, ICAI should include specific learning points in its reviews.

12) ICAI rates the findings from both full and rapid reviews by using a four-tier traffic light system. RAG (Red-Amber-Green) ratings are most effective in dynamic situations, e.g. managing risk, but can be a crude measure for a point in time review, such as ICAI’s reviews. Indeed, two international organisations that are most closely aligned to ICAI in their purpose do not appear to use any ratings: the German Institute for Development Evaluation (DeVal) and Sweden’s Expert Group for Aid Studies (EBA).

13) Rich findings from months of extensive review work, interviews and visits, covering sometimes many years of policy implementation, as well as many hundreds of pages of literature reviews and evaluations, are hard to capture through a single RAG rating. A published scoring methodology would be helpful to ensure a shared understanding of how an assessment is reached and how a ‘green’ rating could be achieved. This would foster a constructive approach to learning rather than focusing attention on what went wrong or did not work. ICAI should move from a single RAG-rated score to a more nuanced rating system that drives change. This may be as simple as rating individual categories of the findings rather than attempting to put one single verdict on what are always a range of complex issues. See Box 2 for suggestions.

Recommendation 7: ICAI should consider developing, in consultation with the Government, partners and the IDC an improved approach to assessment ratings in reviews, by 2023.

Box 2: assessments and ratings

ICAI could consider other organisations’ approaches to audit or assessment. Two international organisations that are most closely aligned to ICAI in their purpose do not appear to use any ratings: the German Institute for Development Evaluation (DeVal) and Sweden’s Expert Group for Aid Studies (EBA). Examples where activities are different to ICAI’s work but may be useful to consider are:

The National Audit Office (NAO) produces a range of outputs and its audit approach depends on the context of each examination. It sets out its evaluative criteria and evidence base in each report’s methodology appendix. When undertaking audits of major programmes, it has developed a common framework which draws together key questions in four areas: Purpose (including need and stakeholder engagements), Value (including business case and cost), Programme set-up (including governance, risk and resources), and Programme delivery and variation management (e.g. performance management and lesson learned). It is a flexible approach that can be tailored, based on issues such as the stage and type of programme. Its value for money reports will contain an overall conclusion on whether the programme is achieving value for money.
The Multilateral Organisations Performance Network (MOPAN) is a network of like-minded donor countries for monitoring the performance of multilateral development organisations. MOPAN assesses organisational effectiveness rather than policy or programmes areas. Each assessment of organisational performance is based on a framework (page 22), which has evolved and is based on MOPAN’s own theory of change. MOPAN uses an indicator framework with four areas (strategic, operational, relationship and performance management) under which are twelve key performance indicators (KPIs) and further micro-indicators (MIs). The detailed methodology is publicly available, and it states that ‘robust analytical strategies must underpin MOPAN indicator ratings to ensure that they are valid and reliable and that the evidence base is credible.’ These are then scored on a scale of 0 to 4. An overall rating is not given, but a detailed assessment is made against these KPIs and shown in a clear diagram. This detailed methodology is more suitable for assessing overall effectiveness of a range of different organisations (often tasked with complex objectives) but is helpful to learn from.

14) ICAI also conducts follow up reviews – these are seen as very helpful – constructive, open, noting progress and acting as a catalyst to continue challenging Government to better implement recommendations. The value of these reviews could be enhanced by additional flexibility in the timing for each review given the different issues in play and lead-in times for implementing recommendations. There is also an opportunity for enhancing learning by bringing together officials, ICAI consultants and other policy experts to discuss progress and consider next steps; recognising that recommendations may need to evolve and ensuring follow up reports do not become a ‘tick box exercise’.

ICAI’s recommendations

15) ICAI has made 348 recommendations since its creation, an average of 4.4 per review. Government has accepted 334 (of which, 98 were partially accepted), and rejected just 14 (of which, 8 have been implemented or are in the process of being implemented)^{[footnote 4]}. This shows the respect and value Government places on ICAI’s role and that the majority of recommendations are deliverable. Even greater impact can be achieved by improving recommendations and ensuring they are both costed and practical and based on academic and empirical evidence on best practice.

16) The success of ICAI’s recommendations depends on Government’s ability to implement them. There are some questions, for example, around the deliverability of some of ICAI’s recommendations and whether they demonstrate sufficient understanding of how Government works, particularly with non-traditional methods / non-grant programmes and ODA spent by departments other than the former DFID. There were also general reflections about the need for a range of recommendations – from very specific, immediately actionable recommendations for programmes, through to systemic / institutional learning recommendations. ICAI should work with departments to target these different types of recommendations to the correct owners, and give a prioritisation reflecting the impact ICAI believes it will have.

17) The World Bank recently published a review of its scrutiny body, the International Evaluation Group (IEG), which included looking at how they could improve the effectiveness of their recommendations. Whilst specific to the IEG, they provide a helpful insight into enhancing the value added of recommendations^{[footnote 5]}.

Recommendation 8: ICAI should consider the recommendations from the World Bank report and continue to reflect best practice in their own recommendations.

Box 3: edited highlights from the recommendations for the IEG (from the World Bank report)

When deciding what to evaluate … give weight to evaluations’ potential to spark change.
Evaluate early experiences … evaluations of early implementation progress can be valuable by providing insights to help … shape an agenda that is growing in importance.
Recommend major changes. Few IEG recommendations call for the Bank Group to stop doing something or to change an approach that was not working.
Write fewer recommendations. Evaluations can sum up lessons and make suggestions but provide fewer recommendations that focus on the major changes that are likely to be impactful.
Continue to make recommendations that are directional yet implementable. This means avoiding recommendations that management may perceive as too prescriptive or too specific on how to achieve the desired outcomes while maintaining a realistic sense of what steps management could plausibly take
Continue to pay close attention to drafting. Some recommendations could have been drafted more clearly, more concisely, or in a more constructive tone. The recommendations and their motivating text … need to be easily understood by readers unfamiliar with the evaluation’s context years after the evaluation is completed.

How Government engages and responds

18) The scrutiny process relies on the positive engagement of the department or body under review and a commitment to transparency and data sharing. The FCDO, and all other ODA-spending departments, needs to be a field-leader on aid transparency.

19) The Government is required to respond to ICAI reviews, and since the 2017 Tailored Review, the aim has been to do this within six weeks. Since June 2013, the average time taken for the Government to respond to a report is 4.5 weeks after publication and 91% of all Government responses are made within 6 weeks of a report’s publication. This suggests that the 6-week timeframe is practicable^{[footnote 6]}. However, the publication of the response (and any subsequent IDC hearing) is not the end of the engagement, and there are a number of ways to improve the relationship to increase the focus on learning and driving impact from ICAI’s reviews.

20) The creation of the FCDO provides the opportunity for better monitoring and integration of ICAI’s findings and recommendations into key institutional structures so that they are considered at the right working levels to make change happen, particularly recommendations aimed at systemic change. This will help embed learning from ICAI’s work into the FCDO’s policy and delivery cycle and ensure that individual reports and related recommendations are not considered in silo. Teams will also benefit from challenge to be positive and ambitious in how they respond to ICAI’s reports.

Recommendation 9: FCDO should determine the best internal committees and reporting structures to monitor recommendation uptake, ensure Government responses are completed on time, and consider systemic issues and wider learning points arising from recommendation 6 – and make that process clear to ICAI.

21) Senior leaders in the FCDO should have increased engagement with ICAI to ensure that aid scrutiny is fully understood in all parts of the new department, and to bring new perspectives to the former DFID-ICAI relationship. The ICAI Chief Commissioner met the former DFID Permanent Under Secretary and Directors-General annually to consider reports, broad lessons arising and progress in implementing recommendations; when invited, ICAI also attended formal cross-Government ODA Senior Officials Groups (attended by Directors), but there has not been a formal, regular relationship with senior officials from other Government departments.

Recommendation 10: The FCDO Management Board will meet annually with ICAI Commissioners, with the option of including Directors-General from key ODA-spending departments as needed. This meeting should focus on learning and reflection on where ICAI reviews are having most impact now and could have most impact in the future.

22) The current process for publishing reports and Government responses can encourage defensiveness. ICAI must retain absolute independence in its reports, but during the process there could be greater confidential discussion of recommendations and how the Government can best tackle the issues which ICAI identifies. ICAI would not be expected to change its recommendations or its reports in any way.

Wider learning

23) A stronger organisational approach to responding to ICAI’s reports and greater co-working to develop positive, action-driven Government responses to ICAI (as outlined above), will be mutually reinforcing for wider learning in FCDO and ICAI. ICAI’s reports are a body of expertise that should be a global public good; the literature reviews that underpin their reports are now published online, which is very welcome. Further value could be driven from ICAI’s work if it was better able to be used by teams seeking to establish a new intervention (see recommendation 6).

24) Ideally, Government would want to tap into ICAI’s expertise beyond the publication of individual reports – both as follow up to specific reports and as a way of scoping new areas, but the current operating model makes this difficult. ICAI should consider additional activities that would enhance its role in learning, linking with FCDO’s own organisational learning systems. This would not involve compromising ICAI’s independence but strengthening the learning system that ultimately leads to improved aid effectiveness. ICAI’s Commissioners should also consider what key best practice / learning they could highlight in their annual report that would have wider application.

Recommendation 11: ICAI and FCDO to consider together how to expand ICAI’s toolkit beyond reviews and reports to include more informal ways of helping organisational learning (e.g. workshops, roundtables, webinars), and how ICAI’s reviews and annual report can capture best practice.

Question 6

Chapter three: how well ICAI does it

Accepted Answer

This section is about ICAI’s efficiency and effectiveness, reflecting on their progress since the 2017 Tailored Review, ICAI’s performance management, and value for money.

ICAI’s response to the 2017 Tailored Review

25) The 2017 Tailored Review concluded that ICAI had effectively scrutinised UK ODA during its first six years and that there was a continuing need for its functions and remit. The review said that ICAI could helpfully prioritise efforts to maximise the impact and influence of its work. It also proposed changes to ICAI’s operating model and noted the importance of ICAI adapting its approach in order to work with departments other than DFID. Following the 2017 Tailored Review, ICAI have improved its operating model by switching to a full-time Chief Commissioner, reducing the Board of Commissioners from four to three, and continuing to monitor and report on the cost of reviews.

26) While ICAI has now implemented the majority of the recommendations from the Tailored Review, similar issues were raised during this review’s consultation with key stakeholders. For example, the review recommended that ‘the priority setting process which informs ICAI’s annual work plan could be strengthened by establishing greater communication with major ODA spending departments.’ This does not mean there has not been progress in these areas, nor that these are intractable problems, but that there is always scope to improve.

How ICAI is held to account / measuring ICAI’s performance

27) Two particularly important recommendations from the 2017 Review relate to how ICAI measures its own performance, namely the creation of a Theory of Change (TOC), and the ongoing development of their set of key performance indicators (KPIs). The TOC is helpful but still in its infancy – we would expect that this will continue to evolve with ICAI, and they may wish to consult TOC experts within the FCDO and elsewhere to help them to develop it.

28) There should be a clear line of sight from the TOC through to the KPIs. KPIs can be useful in driving performance internally but they should also play a key role in how ICAI are held to account for how it delivers its remit and spends public funds. ICAI’s Chief Commissioner is appointed by, and responsible to, the Foreign Secretary and reports on ICAI’s performance to Parliament, through the IDC. ICAI should therefore seek challenge from both Parliament and the FCDO as they continue to develop their KPIs to reflect the recommendations in this review. The KPIs should reflect the stronger focus on learning and must not create unintended incentives. The KPIs can be used by ICAI, Parliament and the senior official lead for ICAI to monitor ICAI’s delivery alongside its annual report.

Recommendation 12: ICAI should continue to develop and adapt its KPIs alongside refocusing its remit in order to give effect to recommendations in this report by March 2021.

29) Government could further drive learning if each Chief Commissioner wrote a no fear or favour report for Ministers once they complete their tenure about what ICAI should do next, including commentary on relationships with Government.

Skills and knowledge of ICAI Commissioners, secretariat and contractors

30) ICAI’s Commissioners have a good blend of skills and backgrounds, in development but also in Parliament, scrutiny, and business, which brings valuable perspectives to scrutiny. Currently Commissioners are all appointed at the same time for the same fixed term – there is some concern about loss of institutional knowledge and delays from inductions of these different incarnations of ICAI caused by simultaneous turnover of Commissioners. The timing of the process for tendering the contract for ICAI’s main supplier is disconnected from Commissioner appointments, which means Commissioners miss the opportunity to be actively involved.

Recommendation 13: Going forward, new Commissioners should continue to be appointed for a single term, but their appointments should be staggered. Timing could also be better sequenced with supplier procurement.

31) ICAI has generally been able to operate more effectively with the former DFID than with other Government departments. ICAI acknowledged it has been harder to establish relationships, and there are questions over the reviewers’ knowledge of other departments, particularly those responsible for non-traditional / non-grant aid programmes. The annual senior officials meeting with ICAI and Directors-General from other Government departments (recommendation 10) would help gain better traction, but ICAI (and its contractors) should also seek further expertise in non-traditional and non-grant aid programmes.

Recommendation 14: To ensure it can review effectively an expanding range of ODA instruments and programmes, ICAI should ensure it has the right skills and expertise, both in house and through its supplier.

Recommendation 15: FCDO to investigate options for secondments of Government staff to the ICAI Secretariat and review teams, and vice versa, putting sufficient conflict of interest protocols in place.

Value for money

32) Since the 2017 Tailored Review, ICAI has sufficiently responded to the Value for Money (VFM) recommendations to improve on cost and operational inefficiencies^{[footnote 7]}. This included bringing some review processes in-house and adjusting the operating model and governance structures. ICAI ensures ongoing cost containment by managing variable admin costs, which has led to a recent underspend. In addition to the 15% per annum reduction in the ICAI travel budget of £36,000 over 4 years, this amounts to an 11.3% reduction in total non-fixed costs. Therefore, ICAI have exceeded the Tailored Review’s suggested 5% pa reduction target on discretionary spend. We agree with ICAI that individual review costs are not appropriate to benchmark due to the variability in costs depending on the review type. However, the day rates used by the supplier were benchmarked by DFID as part of the bid evaluation and were reportedly in line with market rates. This is a sufficient cost control measure.

33) An issue raised repeatedly was the intensive nature of reviews, particularly in drawing on officials’ time – for interviews, country visits, and preparing documentation, as well as occasionally needing to extensively brief reviewers to ensure sufficient knowledge of the policy area, and fact-checking. ICAI has calculated the cost of its reviews but does not include anything beyond the contractor/supplier costs – so the costs do not include Commissioner time, a proportion of ICAI’s secretariat costs, nor crucially the costs of extensive engagement by officials. Scrutiny and the challenge ICAI provides to teams is an integral part of policy development and has intrinsic value and potential to improve policy development and delivery even before reports are delivered. It is therefore important that engaging with an ICAI review is not viewed solely through the lens of cost. Teams should be able to monitor and track the cost of being reviewed but it would be counterproductive to impose a target or cap.

34) ICAI delivers its reviews through contractors, with the in-house Secretariat focusing on operations and support. From July 2019 to March 2020, the costs of the contractor in the production of reviews was around £1.77 million, out of a total spend of around £2.49 million, representing over 70% of the budget^{[footnote 8]}. This spend is based on modules in the service provider contract and the associated day rates. The ICAI secretariat confirmed that the budget for each review is scrutinised as part of the review design phase to ensure it is proportionate to the scope and methodology required.

35) Using contractors in this way has enabled ICAI to bring in specific expertise in response to their workplan. But there may be greater flexibility and value for money to be found through alternative, mixed-source models, e.g. increasing the number of contractors or recruiting some expertise to be in-house. This would help address weaknesses in effectively reviewing a wider range of ODA instruments and programmes. ICAI should investigate options ahead of the next procurement round.

Question 7

Summary of recommendations

Accepted Answer