Roadmap for an evaluation of the National Training Fund

Roadmap for an evaluation of the National Training Fund ELISH KELLY, ELISA STAFFA, ADELE WHELAN, SEAMUS MCGUINNESS AND LUKE BROSNAN ESRI RESEARCH SERIES Number 221, December 2025 ROADMAP FOR AN EVALUATION OF THE NATIONAL TRAINING FUND Elish Kelly Elisa Staffa Adele Whelan Seamus McGuinness Luke Brosnan December 2025 RESEARCH SERIES NUMBER 221 Available to download from www.esri.ie https://doi.org/10.26504/rs221 © 2025 The Economic and Social Research Institute Whitaker Square, Sir John Rogerson’s Quay, Dublin 2 This Open Access work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly credited. ABOUT THE ESRI The Economic and Social Research Institute (ESRI) advances evidence-based policymaking that supports economic sustainability and social progress in Ireland. ESRI researchers apply the highest standards of academic excellence to challenges facing policymakers, focusing on ten areas of critical importance to 21st century Ireland. The Institute was founded in 1960 by a group of senior civil servants led by Dr T.K. Whitaker, who identified the need for independent and in-depth research analysis. Since then, the Institute has remained committed to independent research and its work is free of any expressed ideology or political position. The Institute publishes all research reaching the appropriate academic standard, irrespective of its findings or who funds the research. The ESRI is a company limited by guarantee, answerable to its members and governed by a Council, comprising up to 14 representatives drawn from a crosssection of ESRI members from academia, civil services, state agencies, businesses and civil society. Funding for the ESRI comes from research programmes supported by government departments and agencies, public bodies, competitive research programmes, membership fees, and an annual grant-in-aid from the Department of Public Expenditure, Infrastructure, Public Service Reform and Digitalisation. Further information is available at www.esri.ie. THE AUTHORS Seamus McGuinness is a Research Professor at the Economic and Social Research Institute (ESRI) and an Adjunct Professor at Trinity College Dublin (TCD). Elish Kelly and Adele Whelan are Senior Research Officers at the ESRI and an Adjunct Associate Professors at TCD. Elisa Staffa is a Research Analyst at the ESRI. Luke Brosnan is a Research Assistant at the ESRI. Both Seamus McGuinness and Elish Kelly are Research Fellows at the Institute of Labor Economics (IZA) in the University of Bonn. ACKNOWLEDGEMENTS The work carried out in this report was funded by the Department of Further and Higher Education, Research, Innovation and Science (DFHERIS) as part of its Joint Research Programme with the ESRI on Irish Skills Requirements. We would like to thank all the individuals within DFHERIS who provided assistance during the project, particularly Ashling Abbott, William Beausang, Stephen Ennis, Emma Kinsella, Megan O’Dowd, Anne Ribault and Regina Ward. Valuable contributions were also made by members of the Research Programme Steering Committee: Sorcha Carthy (Higher Education Authority, HEA), Helen McMahon (Enterprise Ireland), Joan McNaboe (SOLAS), Eoin Morton (Skillnet Ireland), Vivienne Patterson (HEA), Aisling Soden (IDA Ireland), Diarmaid Smyth (Department of Enterprise, Trade and Employment) and Emer Smyth (ESRI). We are extremely grateful to the members of this committee for their ongoing support and feedback on the research. We would also like to sincerely thank the representatives from all the grantee organisations that receive National Training Fund (NTF) funding, whose engagement, information provision and participation in interviews and meetings were essential for the completion of this study. This includes colleagues from the Department of Social Protection, Engineers Ireland, Enterprise Ireland, the HEA, IDA Ireland, Skillnet Ireland, SOLAS and The Wheel. We would also like to thank Michelle Foley and David Nevin in the National Apprenticeship Office for their assistance with apprenticeship data queries, and Rob Kelly and Brian Stanley in the Central Statistics Office for their help with a number of queries on the Educational Longitudinal Database. The input that we received from each individual who engaged with us throughout this study’s process was greatly appreciated. This report has been accepted for publication by the Institute, which does not itself take institutional policy positions. All ESRI Research Series reports are peer reviewed prior to publication. The authors are solely responsible for the content and the views expressed. Table of contents | i TABLE OF CONTENTS ABBREVIATIONS ...................................................................................................................................... v EXECUTIVE SUMMARY .......................................................................................................................... vii CHAPTER 1 Introduction ......................................................................................................................... 1 1.1 Background context ................................................................................................................... 1 1.2 Objectives of this report ............................................................................................................ 4 1.3 Methodology .............................................................................................................................. 6 1.4 Structure of the report ............................................................................................................... 7 CHAPTER 2 Overview and context .......................................................................................................... 8 2.1 Introduction ............................................................................................................................... 8 2.2 Analysis of NTF expenditure ...................................................................................................... 8 2.2.1 Labour market initiatives .................................................................................................. 8 2.2.2 Firm-level initiatives ........................................................................................................ 10 2.2.3 Social/Other initiatives ................................................................................................... 12 CHAPTER 3 Literature ............................................................................................................................ 13 3.1 Introduction ............................................................................................................................. 13 3.2 International literature on National Training Funds ................................................................ 13 3.2.2 Training fund effectiveness ............................................................................................. 14 3.3 Impact evaluation for general active labour market policy training programmes .................. 17 3.3.1 International literature ................................................................................................... 17 3.3.2 Irish context..................................................................................................................... 20 3.3.3 Summary of NTF-funded initiative evaluations .............................................................. 24 CHAPTER 4 Evaluation possibilities for the National Training Fund ..................................................... 28 4.1 Introduction ............................................................................................................................. 28 4.2 Labour market initiatives ......................................................................................................... 29 4.2.1 Datasets that facilitate CIE evaluation of NTF-funded initiatives ................................... 31 4.2.2 Datasets with significant gaps and challenges to evaluation .......................................... 34 4.3 Firm-level initiatives ................................................................................................................. 42 4.3.1 Significant gaps and challenges to evaluation ................................................................ 43 4.4 Social and other initiatives ....................................................................................................... 49 ii | Roadmap for an evaluation of the National Training Fund 4.4.1 Suitable for monitoring ................................................................................................... 52 4.5 NTF evaluation feasibility analysis ........................................................................................... 52 4.6 Summary .................................................................................................................................. 56 CHAPTER 5 Summary and conclusions ................................................................................................. 58 REFERENCES .......................................................................................................................................... 61 APPENDIX I Information gathering template issued to NTF grantee organisations ............................. 65 APPENDIX II Overview of NTF-funded initiatives .................................................................................. 69 Executive summary | ix the scale of NTF funding, these are all issues that need to be addressed, including the gaps in firm-level data, to ensure that the effectiveness of the fund’s initiatives can be monitored and assessed. The report proposes prioritising evaluation of the largest labour market initiatives where CIE is currently feasible, while taking steps to strengthen data infrastructure for programmes that cannot yet be fully evaluated. Improved data collection, access, research capacity and evaluation skills will be essential to fully leverage Ireland’s administrative data, and to ensure the effective use of NTF resources. The data conditions for rigorous evaluation are now increasingly in place. With appropriate prioritisation and further progress with data systems, this can help the NTF to deliver maximum value for individuals, businesses and the wider economy. 1 | Introduction CHAPTER 1 Introduction 1.1 BACKGROUND CONTEXT The National Training Fund (NTF), established under the 2000 National Training Fund Act, was set up to support employment-focused training. In particular, the goal of the fund is to support schemes that raise the skills of those currently in employment and provide training to those seeking employment. A small amount of the fund is also allocated to researching skill requirements in the economy, both existing and future. 2 In 2023, the fund was supporting 18 initiatives, 3 which were under the remit of the following grantee organisations: i) the Higher Education Authority (HEA); 4 ii) SOLAS; 5 iii) Skillnet Ireland; 6 iv) the Department of Social Protection (DSP); 7 v) Enterprise Ireland; 8 vi) IDA Ireland; 9 vii) Engineers Ireland; 10 and viii) The Wheel. 11 The NTF is managed by the Department of Further and Higher Education, Research, Innovation and Science (DFHERIS). Some of these 18 initiatives consisted of a number of sub-programmes (see Figure 1.1 and Tables 2.1 and 2.2). The NTF is funded through a levy on employers that is collected through pay related social insurance (PRSI). 12 Up until 2017, the rate of levy was 0.7 per cent. In 2016, the Expert Group on Future Funding for Higher Education recommended that the levy be increased, as a means by which employers could contribute to the 2 It has been proposed that, going forward, a proportion of the NTF will be directed towards funding capital infrastructural projects (i.e., equipment). 3 For further information on these 18 initiatives, please see Figure 1.1 and its supporting notes. 4 The HEA is a state agency tasked with governing and regulating the higher education system, and its institutions, in Ireland. 5 SOLAS is a state agency charged with overseeing the further education and training sector in Ireland. 6 Skillnet Ireland is a business support agency of the Government of Ireland. Its primary objective is to advance competitiveness, productivity and innovation of businesses operating in Ireland through enterprise-led workforce development. 7 The DSP is Ireland’s government department that promotes active participation and inclusion in society among those not in employment (e.g., the unemployed, lone parents, pensioners) through the provision of income supports, employment services and other services. 8 Enterprise Ireland is an Irish government enterprise development agency tasked with supporting Irish businesses operating in the manufacturing and internationally traded services sector. 9 IDA Ireland is a statutory investment promotion agency that assists multinational companies to grow and expand into Ireland. 10 Engineers Ireland is the representative body for the engineering profession in Ireland. 11 The Wheel is Ireland’s national association of charities, community groups and social enterprises. They provide leadership, along with advice, training and other opportunities, to people working or volunteering in the charity and community sector. 12 The levy is imposed on Class A and Class H PRSI employees: these make up 75% of all insured employees. Roadmap for an evaluation of the National Training Fund |2 increase in higher education funding. As a result, the rate increased to 0.8 per cent in 2018, 0.9 per cent in 2019, and since 2020 it has been 1 per cent. Income allocated to the NTF has increased significantly in recent years. In 2017, the annual income was €436 million. By 2022, however, the annual income had increased to €951 million, and to €1,080 million by 2023. 13 The increase in the levy rate is part of the reason for this. However, as the fund’s income tends to be procyclical, over the last few years the main driver for the rise in the fund’s income has been the robust performance of the economy, with greater numbers of people in employment and increases in earnings. 14 Specifically, unlike after the Great Recession, the economy rebounded well after the COVID-19 health pandemic, and it is continuing to perform strongly, with modified domestic demand (MDD) forecast to be 3.8 per cent in 2025 and 2.9 per cent in 2026. 15 The numbers in employment are also expected to grow, increasing from 2.757 million in 2024 to 2.813 million in 2025 and to 2.858 million in 2026 (Barrett at al., 2025). The implications of this for the NTF is likely to be the continued growth in the availability of funds to support those initiatives focused on raising the skill levels of those currently in employment, as well as programmes that will provide employees and graduates with the skills required to meet future labour market needs. With our current unemployment rate standing at 4.6 per cent (Barrett at al., 2025), there is currently a lower share of NTF funding being allocated to initiatives that are supporting those seeking employment. Nevertheless, after a period of expansion, economies naturally go through a contraction (a recession). Recognising this is important for ensuring that the NTF remains flexible in how it allocates resources, particularly to initiatives that support both employed individuals and jobseekers. The issue of skill shortages has been highlighted in The Government response to Ireland’s competitiveness challenge 2024, 16 and, more widely for Europe, in the Draghi report: A competitiveness strategy for Europe. 17 Presently, the NTF has an accumulated surplus. This stood at €1.54 billion at the end of 2023, and is 13 NTF income information for 2023 was provided by DFHERIS. The NTF expenditure figures for these three years were: €357 million (2017), €681 million (2022) and €909 million (2023). Estimated expenditure of the NTF is included within the expenditure ceiling of DFHERIS. Given this, increases in NTF spending need to be considered within budgetary policy and EU fiscal spending rules. For further information, see: 2025-09-23_general-scheme-briefing-paper-national-training-fundamendment-bill_en.pdf [accessed 16 October 2025]. 14 The fund’s income tends to be pro-cyclical, increasing during periods of high employment and decreasing when employment falls. 15 Similar to GDP, modified domestic demand (MDD) is a measure of domestic economic activity. Specifically, MDD covers personal and government consumption, and modified investment (i.e., total investment less leased aircrafts and imports of research and development and intellectual property from intangibles). MMD is a smaller number than GDP, and is seen to more accurately reflect how households, the Government and domestic corporations in Ireland are doing. For further information on MDD see, for example, CSO, ‘Total domestic demand and modified total domestic demand.’ 16 See the-government-response-to-ireland-s-competitiveness-challenge-2024.pdf [accessed 15 September 2025]. 17 See The Draghi report on EU competitiveness [accessed 15 September 2025]. 3 | Introduction estimated to have been €1.77 billion at the end of 2024. 18 While some of this surplus has been earmarked for a funding package for the tertiary education sector, for the period 2025 to 2030, 19 given the aim of the NTF, its income can also be considered for use to support initiatives that will help address the skills shortages issue. 20 While the employment permit system was expanded in 2023 to assist in addressing Ireland’s skills shortages issue, shortages remain. 21 NTF-funded initiatives could help to address this issue by focusing on provision of education and training needed to address the identified shortages. A review of the NTF was carried out in 2018 (Indecon, 2018). This review identified four key areas of reform, one of which was ‘improvements in monitoring and evaluation of NTF’. 22 It made 14 detailed recommendations across the four areas. An implementation plan outlining a number of reforms, which were to have been executed to support two 0.1 per cent rises in the NTF levy that were part of Budget 2019, was developed and published by the Department of Education and Skills in 2019. Three of the reforms within this plan focused on improving monitoring and evaluation of the NTF. Specifically: • The Department would organise and publish an NTF evaluation report on an annual basis, to include counterfactual modelling of the programmes’ impacts. • Performance metrics should be expanded to support enhanced monitoring of outcomes of all NTF-funded programmes. Metrics should include measures that track progression outcomes; for example, to employment, educational progression (including certification achieved), employment placement and sustainment, and completion rates. • Priority should be given to drive continued enhancement of data to inform evaluation of the NTF. An examination of this implementation plan by the Comptroller and Auditor General in 2022 found that planned reforms had been implemented in three of the four key areas, but that identified reforms around ‘improvements in monitoring 18 See Houses of the Oireachtas (2025). ‘Education and training provision’, Parliamentary Questions, 34th Dáil, 22 January. 19 See DFHERIS (2025). ‘Minister Lawless obtains Government approval to amend National Training Fund Act, unlocking funding package of €1.5 billion’, press release, 29 April [accessed 15 September 2025]. 20 Given that NTF spending is governed by departmental spending rules, specifically falling within the expenditure ceiling of DFHERIS, this effects how much of the NTF surplus can be spent annually on initiatives that support the goals of the NTF. This has resulted in some organisations recommending the removal of NTF funding from state spending rules. For further information, see: An overview of the National Training Fund (NTF) and 2025-09-23_general-scheme-briefing-paper-national-trainingfund-amendment-bill_en.pdf [accessed 16 October 2025]. 21 At the time of writing, the employment permit system, and list of occupations it covers to address skill and labour gaps, is being reviewed; See DETE (2025). ‘Ministers Burke and Dillon initiate public consultation on review of employment permit occupations lists’, Department News, 23 July. 22 The other three areas were: i) reform of the future direction of the NTF; ii) utilising NTF to support investment in higher education; and iii) enhancing enterprise engagement and input to NTF priorities. Roadmap for an evaluation of the National Training Fund |4 and evaluation of NTF’ had not progressed in line with the timelines that had been set out in the implementation plan. Given this, as part of the DFHERIS joint research programme with the Economic and Social Research Institute (ESRI), the Department asked the ESRI to develop a framework to monitor and assess the effectiveness of the various NTF-funded initiatives. Following extensive engagement with all grantee organisations in receipt of NTF resources on their data infrastructure for their NTF funded initiatives, this report sets out the ESRI’s proposed framework for evaluating the NTF. 1.2 OBJECTIVES OF THIS REPORT The aim of this study is to assess the extent to which the various NTF-funded initiatives can be readily evaluated and, on the basis of this assessment, to develop a framework for DFHERIS to use to evaluate the NTF. This was undertaken by: assessing the nature and objectives of each NTF intervention; deriving relevant outcomes – key performance indicators (KPIs); and then identifying and examining the data that are currently held by the NTF grantee organisations and assessing whether they meet evaluation requirements. In particular, we concentrate on the feasibility of conducting counterfactual impact evaluations (CIEs) for the various NTF interventions. 23 Under this evaluation approach, those who engage in an NTFfunded initiative (i.e., the treatment group) would be compared with those individuals with similar characteristics that do not participate in the training programme (i.e., the control group) on a key objective/outcome of the programme. For example, the rate of employment of supported job seekers (the treatment group) might be compared with a comparison group not in receipt of such supports (the control group). Given this methodological approach, we categorise each NTF initiative as follows: • suitable for evaluation; • partially suitable for evaluation; or • facing data gaps and challenges that would hamper evaluation. Where CIE is not feasible, we examine whether the data infrastructure will allow for the use of monitoring tools to track the programme’s effectiveness across a set of relevant KPIs. In this study, the NTF-funded initiatives, which are discussed in detail in subsequent chapters and accompanying appendices, are grouped into the following three categories, based on our assessment of their core objectives: 23 A CIE involves quasi-experimental methods, which are used to estimate the causal impact of a policy, programme or intervention; for example, the effectiveness of an unemployment training programme in terms of helping unemployment recipients to re-integrate into employment. Quasi-experimental methods include propensity score matching (PSM), difference-in-differences, regression discontinuity design and instrumental variables. For further information, see, for example, Imbens and Wooldridge (2009). 5 | Introduction • Labour market; • Firm level; and • Social/Other. A number of these initiatives are also cross-cutting, in that while their primary objective might be, for example, labour market focused, there are also spillover effects for one of the other two areas, such as secondary firm-level effects. The various initiatives examined in this report, which are based on 2023 NTF expenditure, are set out in Figure 1.1. 24 24 Some NTF funding is allocated to institutions that are in charge of providing information on skill requirements, such as the Skills and Labour Market Research Unit within SOLAS, the Expert Group on Future Skills Needs within DETE and Regional Skills Fora and Regional Skills Innovation within the Skills Planning and Enterprise Engagement Unit. Our study does not include analysis of this type of NTF funding. Roadmap for an evaluation of the National Training Fund |6 FIGURE 1.1 2023 NTF INITIATIVES Source: Constructed by the authors using information from the Comptroller and Auditor General’s 2022 report on the accounts of the public service. Notes: (1) captures ‘labour market/firm’ cross-cutting initiatives, and (2) ‘labour market/other/social’ cross-cutting programmes. * Consists of seven sub-programmes; ** consists of three sub-programmes; *** consists of two main programmes, one of which consists of nine sub-programmes. 1.3 METHODOLOGY A combination of desk-based research, consultations and semi-structured interviews were employed to undertake this study. Specifically, we began by examining the international literature to identify how funds similar to the NTF have been evaluated in other countries, including, more broadly, studies that have evaluated labour market programmes and firm-related training initiatives similar to those funded under the NTF (see Chapter 3). Labour market: SOLAS: Training People For Employment* SOLAS/NAO: Apprenticeships (1) HEA: Enterprise-Focused Higher Education Provision (1) HEA/NAO: Apprenticeships (1) HEA: Springboard+ For Employment HEA: Springboard+ In Employment SOLAS: Employee and Continuing Professional Development (CPD)** HEA: Human Capital Initiative - Pillars 1 and 2 (1) DSP: Community Employment Training (2) Skillnet Ireland: Training Programme  Seeking Employment DSP: Training Support Grants SOLAS: Traineeship (1) DSP: Work Placement Experience Programme Social/Other: HEA: Human Capital Initiative  Pillar 3 (Other) The Wheel: Community and Voluntary Organisations (Social) Firm: Skillnet Ireland: Training Programme  Employment (1) Enterprise Ireland: Training Grants to Industry*** (1) IDA: Training Grants To Industry (1) Engineers Ireland: Employee and Continuing Professional Development (1) (2) (1) 7 | Introduction From this work, we developed a template that sets out a range of questions to capture information on the data infrastructure in place for each NTF-funded initiative (see Appendix I). This was administered to the grantee organisations in July 2024, for them to complete. Consultation meetings were also offered and arranged with a number of the grantee organisations at this time to: • explain the research being undertaken; • provide the rationale for the information in the template that we were requesting for them to complete; and • address any questions and/or concerns that they had about the research. The majority of the grantee organisations completed and returned their templates during August 2024; others returned theirs in September 2024 and January 2025. On reviewing the information provided in the templates, follow-up queries were sent to a number of grantee organisations at the end of October, with returns received by mid-November. Semi-structured interviews took place online with a range of grantee organisations, between December 2024 and February 2025, in order to clarify some outstanding queries about their NTF initiative data infrastructures. Specifically, these meetings were held with the HEA, SOLAS, Skillnet Ireland and the National Apprenticeship Office (NAO) (see Table 4.1). We also met with the Central Statistics Office (CSO) to discuss their administrative Educational Longitudinal Database (ELD), a database that we identified as having the potential to evaluate a number of the NTF-funded initiatives (also see Table 4.1 in Chapter 4). 25 Before completion of the report, grantee organisations were asked to review a final draft, to ensure that the information provided within the report on their initiative(s) was factually correct. At that time, final clarifying meetings were arranged with some of the grantee organisations; specifically, The Wheel, Enterprise Ireland and Engineers Ireland. 1.4 STRUCTURE OF THE REPORT The remainder of this report is structured as follows. Chapter 2 provides additional information on the NTF-funded initiatives, specifically regarding spending on the various initiatives, from their inception up to 2023. 26 Chapter 3 provides a summary of the international and national literature on the evaluation of funds similar to the NTF in other countries, and also on the impact of general training programmes on labour market and firm-level outcomes. In Chapter 4, we outline evaluation possibilities for the NTF, while conclusions are outlined in Chapter 5. 25 This link provides information on the CSO’s role in delivering high quality data, such as the ELD, for research and policy making, and how it can facilitate access to these data: https://www.cso.ie/en/trusttransparency/ourroledeliveringbetterdata/. 26 2023 is the most recent year for which we have finalised NTF spending data, provided by DFHERIS. Overview and context |8 CHAPTER 2 Overview and context 2.1 INTRODUCTION In this chapter, we set out the initiatives that were funded by the National Training Fund (NTF) in 2023. In particular, we analyse expenditure on these initiatives from 2015, 27 or from the year they began to receive NTF funding, up to 2023. 28 2.2 ANALYSIS OF NTF EXPENDITURE In Tables 2.1 to 2.3, we set out the various initiatives that were being funded by the NTF in 2023, classified by which of these categories their core objectives fell into: labour market, firm-level and ‘social/other’ focused. Additional detail on each initiative is provided in Appendix II, relating to: the nature of the programme; its objectives (e.g., reskilling); targeted support group (e.g., unemployed); key outcomes (e.g., employment); data collected; data requirements for counterfactual impact evaluation (CIE); and assessment of its potential for a CIE. 2.2.1 Labour market initiatives In 2023, 13 of the 18 initiatives funded (partially or fully) by the NTF had a labour market focus, either to upskill or reskill those in employment or move those in unemployment closer to the labour market or into employment. Five of these initiatives were under the remit of the Higher Education Authority (HEA), four were under SOLAS, three were under the Department of Social Protection (DSP), and one was under Skillnet Ireland. In 2023, these 13 initiatives accounted for 87.4 per cent of the NTF budget. The three largest of these initiatives, based on spending from 2015 (or the programme’s inception year if that came after 2015) to 2023, were SOLAS’s Training People for Employment and Apprenticeship programmes, and the HEA’s Enterprise-Focused Higher Education Provision (Table 2.1). Between 2015 and 2023, almost €1.7 billion was spent on the Training People For Employment initiative, over €900 million on Craft Apprenticeships, and almost €790 million on Enterprise-Focused Higher Education Provision. 27 The year 2015 is taken as the starting point, as the source of the NTF expenditure data outlined in this chapter is the Comptroller and Auditor General’s 2022 NTF review report, and 2015 is the first year for which NTF expenditure information is available in this report. 28 Most recent NTF funding data, provided by DFHERIS. 15 | Literature where employees received training, compared to similar firms whose employees did not attend any training. UNESCO (2022) found that only a small number of CIEs have been carried out on training funds since 2010; these studies use quasi-experimental methods and data from the 2000s or earlier. Most of the literature highlights a generalised lack of robust, quantitative evidence on the causal impact of training funds: this scarcity is due to data collection costs, as well as methodological issues related to correctly identifying suitable control groups. Among national training funds, the Québec (Canada) Workforce Skills Development and Recognition Fund, funded by the university-level business school HEC Montréal, has undergone an impact evaluation. Impact evaluations also have been carried out on the following sector training funds: SENAI in Brazil, a study funded by the UK’s Department for International Development; Fondirigenti in Italy; the Malaysian PSMB, funded by the World Bank; and sector training funds in the Netherlands. Most training funds rely only on output data, or on ‘success’ stories. About onequarter of national training funds and a few sectoral funds analysed by UNESCO (2019) implement tracer studies of graduates of training interventions. These studies follow trainees over time and report some labour market outcomes; however, the data cannot be analysed to evaluate any sort of causal impact of the training, as an adequate control group cannot be identified. A small number of national training funds and some sectoral funds measure their outcomes through perception studies. These can be employer surveys, which ask respondents about their perception of the training programmes, or of the impact of the training on the workforce and company productivity. They can also be surveys of trainees/employees, who are asked about their perception of training programmes and the impact on their career. Through use of open-ended questions, surveys can be used to gather descriptive data that help provide a more qualitative insight into training outcomes, although not any evidence of causal relationship. Research on the impact of training funds attempts to evaluate the extent to which training programmes had any individual/labour market level effect or firm level effect. Section 3.2.2.1 presents a small number of studies that analyse impacts at the individual/labour market level, while Section 3.2.2.2 presents some studies that analyse impacts at the firm level. 3.2.2.1 Labour market outcomes At an individual level, the aim is to assess whether training funds improve individual employability, allow transition from unemployment to employment, and improve access to training for disadvantaged and vulnerable groups. Only the Roadmap for an evaluation of the National Training Fund |16 sectoral training fund SENAI, in Brazil, has had a CIE. SENAI is the Brazilian system of vocational training in the industrial sector: it has a levy-based system, and all industrial companies pay a 1 per cent tax on all payrolls, which funds general training in training centres. SENAI also sells ad hoc training courses to specific companies. Using survey data from the 2007 PNAD Brazilian household survey , Barria and Klasen (2016) evaluate whether past participation to SENAI training had any significant impact on labour market outcomes (i.e., monthly earnings, monthly hours of work, hourly earnings and the probability of being employed in the formal sector) for those trained through SENAI relative to those trained in other training institutions. An inverse probability weighting model was implemented in order to estimate the average treatment effect on the relevant outcomes. The authors found that the average training premium in terms of monthly earnings for those aged 15–29 trained through SENAI was significantly higher compared to those trained in other training institutions (28 per cent versus 10 per cent wage premium). The effect was found to be stronger for males and individuals in rural areas, and it originated from increased hourly earnings rather than monthly working hours. 3.2.2.2 Firm-level outcomes Few studies have evaluated the firm-level effects of training funds. Aspects to assess include whether training funds incentivise companies to raise the incidence of training and whether training improves workers’ productivity within companies. There is no consenus on the impacts of such funds, as some studies found no impact while others registered a small but significant positive impact. In 1995, a law was passed in Québec in Canada that obliged all companies to devote 1 per cent of their payroll towards training; in this ‘train or pay scheme’, firms spending less than 1 per cent had to remit the difference to the government. In 2004, a reform exempted medium-sized companies from the training requirement. Following this reform, Dostie (2012) employed a difference-in-difference strategy to compare changes in training and productivity in medium-sized firms (the treatment group) relative to the small and large firms (the control group). To do so, they accessed a longitudinal, linked employer–employee dataset with detailed information on the training policies of firms, in particular the number of workers undertaking classroom and on-the-job training in a given year. They found that the repeal of the obligation for training had no impact on firms’ training levels, but rather caused firms to change their training ‘allocation’ decision, substituting other forms of training (on-the job training) for the specific type of training required by the law (classroom training). The study concludes that the levy exemption schemes did not successfully raise firms’ training levels. In the Netherlands, sectoral training funds are financed by a levy on the firms’ total payroll costs. Firms can subsequently apply for subsidies for training costs. By employing different datasets on firms’ characteristics, training investments, 17 | Literature workforce characteristics and economic sectors, Kamphuis et al. (2010) compared the level of training investments of firms in sectors with and without a training fund. The authors used a propensity score matching approach, where the treatment group was made up of companies in sectors with a fund, and the control group of those in sectors without a fund. Results showed that training levels were not higher in sectors with a fund than they were in sectors without a fund, so no evidence was found of a significant impact of the training fund on firms’ level of training. Feltrinelli et al. (2017) carried out an evaluation of Fondirigenti, one of Italy’s interprofessional training funds that is concerned with the continuous training of middle managers. The evaluation was based on a panel dataset covering all sectors of the Italian economy over the period 2006–2011, and including information on the middle management training activity of Italian firms and general firm-level indicators. By employing an IV-GMM model, the authors found that training investment devoted to middle managers had a positive and significant effect on firm performance, measured in terms of total factor productivity. 35 Returns on training investments seem to be much higher for large and medium firms, and were found to be not significant for smaller firms. In Malaysia, since 1993, firms in specific sectors have to pay a monthly levy of 0.5 to 1 per cent on the payroll to fund the Human Resource Development Training Fund. The World Bank (2017) found that registration on to this training fund increases a firm’s likelihood of providing training by 24 percentage points and increases the share of workers trained by 19 percentage points. They also found that training an additional 1 percentage point of the workforce is associated with a nearly 1 per cent increase in productivity among all firms in Malaysia, but with a nearly 3 per cent increase in productivity among firms registered on this fund. 3.3 IMPACT EVALUATION FOR GENERAL ACTIVE LABOUR MARKET POLICY TRAINING PROGRAMMES 3.3.1 International literature National training funds involve a levy being placed on private employers, which is used to fund training programmes and activities for those who are employed and unemployed, as well as for firms. However, training programmes are one of the public measures of active labour market policy (ALMP). Active measures, which include job placement, assistance and employment services, training programmes, employment subsidies and direct 35 An instrumental variables–Generalised method of moments model is used in econometrics to estimate relationships between variables when there might be some bias or endogeneity issue. Roadmap for an evaluation of the National Training Fund |18 employment provision, are intended to assist unemployed people to return to work (Kelly et al., 2011). Training tends to account for the largest share of spending on active measures (Martin, 2000); training programmes aim at enhancing jobseekers’ human capital and their employment perspectives. Within the literature, evidence from CIEs of the performance of public training programmes is mixed, even when long-run effects of training are analysed (McGuinness et al. 2011). 3.3.1.1 Labour market outcomes Several studies have found positive effects of participation in training programmes at an individual level. Card et al. (2010) conducted a meta-analysis of 199 programme estimates for active labour market policy evaluation from 97 studies conducted between 1995 and 2007. The authors found that classroom and on-thejob training programmes often improved employment probabilities and earnings for participants in the medium term. The impact is found to be insignificant, and negative when significant, in the short run. The international evidence base pointed towards specific skills training as one of the most effective means of labour market activation. Card et al. (2018) performed a second meta-analysis, and extended the number of studies used as sample in their earlier study (from 97 to 207, conducted before 2014). Again, the authors found that classroom and on-the-job training programmes had a positive impact on the probability of employment for participants in the medium term (1–2 years post programme), as well as longer term. Moreover, the average impacts of ALMPs were found to vary across socioeconomic groups: specifically, several studies found that long-term unemployed participants benefit relatively more from training programmes than from other ALMPs. 36 Utilising administrative data from the Spanish Public Employment Service for 2018–2019, Arranz and García-Serrano (2024) investigated the impact of participation in job search assistance and training programmes on the probability, for unemployed jobseekers, of transition to employment, by using propensity score matching techniques. The study found that, compared to non-participants, participation in these programmes significantly increased the probability of transitioning from unemployment to employment, particularly into jobs of intermediate quality, and training programmes were identified as the primary driver of improved employment outcomes, with a persisting positive effectiveness over time. 36 Other studies exist that provide an overview of European ALMP programmes (such as Kluve, 2010; Caliendo and Schmidl, 2016), and there are studies on the impact of vocational education and training programmes (e.g., Eichhorst et al., 2015). 19 | Literature Recently, the OECD (2025) published results from a wide, joint OECD–European Commission project, conducted between 2021 and 2025, on CIEs of some ALMPs across six European countries (Finland, Greece, Ireland, Lithuania, Portugal and Slovenia). The ALMPs, among them training programmes, were evaluated using linked administrative and survey data and several CIE techniques. Training programmes in Finland, Greece and Lithuania were found to have positive effects on the employment probability of jobseekers in the long term, although negative effects were found in the short term, due to ‘lock-in’ effects. However, impacts were found to vary by socio-economic groups: for instance, the positive effects of training on employment were higher for women and jobseekers over 50 in Finland, and for young people in Greece. In Denmark, a few studies have found insignificant or negative employment effects on participants in labour market training programmes. Rosholm and Skipper (2003), evaluating a specific programme for unskilled workers, found that it increased the subsequent unemployment rates of its participants. Jespersen et al. (2008) found no significant effect, in the short or long term, of classroom training on participants’ earnings or employment. In Sweden, Calmfors et al. (2001), summarising the results of various studies about the impact of public training programmes conducted between 1980 and 2000, found that outcomes varied by their time period. While evaluations of training acquired during the first half of the 1980s suggest positive effects on participants’ employment and/or income, evaluations of labour market training outcomes in the early 1990s found insignificant or significantly negative effects. In an evaluation of six Swedish active labour market programmes, Sianesi (2008) found that individuals joining labour market training displayed lower employment rates in both the short and long term. In Switzerland, Lalive et al. (2008) found that training programmes produced an increase in unemployment duration. Other negative effects were found in several studies in European countries (see Kluve, 2006, for a comprehensive metaanalysis). Insignificant or negative performance of training on individual level outcomes may be due to ‘lock-in’ effects: people who attend training courses have less time available to look for a job and put less effort into job searching, thereby becoming ‘trapped’ in the training programme (Conny, 2016). Alternatively, it may be due to selection bias problems: people who attend training programmes are already experiencing more disadvantaged conditions or greater marginalisation, and therefore experience difficulties in finding job opportunities or are not so active in job searching. 3.3.1.2 Firm-level outcomes Firms invest in training their workers, in the expectation of it leading to enhanced productivity, competitiveness and profitability. However, measuring the return on this investment is challenging, and only a few studies have tried to do so. Problems relate to difficulties in obtaining data: firm-level datasets that have information on Roadmap for an evaluation of the National Training Fund |20 productivity as well as workers’ participation in training are rare (Fialho et al., 2019). Martins (2021) is one of the first studies to provide evidence on the effects of employee training on firm performance. They describe their analysis as being based on almost potentially quasi-experimental variation. Using data from a training grants programme in Portugal, which was supported by the European Social Fund (ESF) of the EU, they compared firms in Portugal that received a grant with unsuccessful grant applicants. Following both the successful and unsuccessful grant applicants, the study boasts an extraordinarily rich matched employer– employee panel dataset. By using a difference-in-differences methodology, the author finds that the additional training driven by the programme led to economically and statistically significant improvements in several dimensions of firm performance. Sales, gross value added (defined as output minus intermediate consumption), employment, labour productivity (measured by the ratio of sales and the number of employees) and exports are all shown to increase among those firms that received the training grant (compared to the control group of unsuccessful applicants). 3.3.2 Irish context 3.3.2.1 Labour market outcomes Compared to other OECD countries, there is a shortage of rigorous evidence on the impact of training in Ireland (McGuinness et al. 2014). Again, evidence from CIEs on the performance of training programmes is mixed. Several studies have found positive effects of training on labour market outcomes for unemployed individuals (McGuinness et al., 2011 & 2019) and for those returning to education (McGuinness et al., 2019). We summarise the main findings below. McGuinness et al. (2011) provided a systematic evaluation of the impact of activation measures implemented under the Irish National Employment Action Plan. Under the plan, persons in receipt of Jobseeker’s Benefit who reached three months duration on the Live Register of unemployment were referred to FÁS, the national training and employment authority, for an activation interview. At this, individuals could be provided with job search assistance, and some may be referred to employment or training opportunities. By using a PSM approach, the authors found that the job search assistance was totally ineffective; however, those who participated in training were less likely to be unemployed over a 21-month time horizon. Moreover, high-level specific skills training was found most likely to increase the probability of participants exiting from unemployment. McGuinness et al. (2014) estimated the differential impact of different types of public training programmes. By employing a propensity score matching (PSM) estimation framework, they found that those who participated in training were less 21 | Literature likely to be unemployed at the end of the two-year study period, but the average effect of training varied by the type and duration of training received. Job search training and high-level specific skills training were most likely to increase the probability of participants exiting from unemployment, while a more modest positive effect was found for general vocational skills programmes, and weak effects were found with respect to low level skills training. Lower duration training programmes had a more positive impact, with the exception of high-level skills training programmes. McGuinness et al. (2019) produced an evaluation of the Post Leaving Certificate (PLC) programme, which is the most significant full-time further education and training programme (post-secondary non-tertiary) serving multiple purposes such as preparing for employment, bridging to higher education and offering secondchance education to adults. By using data from a specially designed learner survey and a PSM approach, the authors compared employment and educational outcomes of PLC participants against individuals who entered the labour market directly after completing their Leaving Certificate (upper secondary). The results showed that PLC education had a strong positive influence on future labour market outcomes: PLC participants were 16 percentage points more likely to be in employment relative to the control group. Moreover, PLC participants were found 27 percentage points more likely to transition to higher education studies. Studies of other interventions found negative effects. Kelly et al. (2022) evaluated Ireland’s second-chance education opportunity scheme, the Back to Education Allowance (BTEA), which was one of the main ALMPs for tackling unemployment. Individuals could participate in a course of education if unemployed while continuing to receive an income support payment. By using the Jobseekers Longitudinal Dataset, an administrative dataset compiled by the Department of Social Protection (DSP), alongside PLC survey data, and employing a PSM technique to compare outcomes of BTEA participants with an adequate control group, the authors found a negative employment impact of the education scheme: BTEA participants were substantially less likely to be in employment between four and six years following entry into their respective BTEA programmes, compared to the control group. Research on the impacts of traineeships on labour market outcomes in Ireland is extremely limited. 37 JobBridge was a national internship scheme launched by the Government in 2011 and operated by the DSP. Indecon (2013) conducted a programme evaluation for individuals who completed the traineeship: they found that total employment rates for JobBridge participants (treatment group) were much higher compared to non-JobBridge participants who exited the Live Register 37 Traineeships are generally considered as having a primary focus on individual-level outcomes, although they may have a secondary focus on firm-level objectives. Roadmap for an evaluation of the National Training Fund |22 (control group) over the same period, with almost two-thirds of JobBridge participants employed within five months of completing the programme versus one-third; however, the educational levels of JobBridge candidates were also very high, making it difficult to separate out the impact of the training received under JobBridge on employment probabilities. In the study, there was a weak attempt to estimate a counterfactual for JobBridge, but this fell well below acceptable international standards. The control group was considered not adequate due to differences between the profile of Live Register participants and that of JobBridge participants. 3.3.2.2 Labour market and social outcomes For some programmes designed to assist unemployed persons back into employment, a purely empirical approach centred on estimating a counterfactual is not sufficient. This is particularly the case for employment supports implemented within the realm of community development, as these supports target individuals facing more substantial barriers to employment (such as physical or mental health problems, language difficulties, etc.). Such supports, which typically target individuals who are deemed to be further (or furthest) away from the labour market relative to typical claimants, are termed pre-employment programmes. Relative to more mainstream labour market activation programmes, the evaluation of pre-employment supports is a more complex exercise, for a number of reasons. Whelan et al. (2019) looked at the Social Inclusion and Community Activation Programme (SICAP) in Ireland and examined how programme impacts could be effectively measured. SICAP represents a major component of Ireland’s community development strategy, led by the Department of Rural and Community Development and the Gaeltacht. The vision of SICAP is to improve the opportunities and life chances of those who are marginalised in society, experiencing unemployment or living in poverty through community development approaches, targeted supports and interagency collaboration. After evaluating the international literature, the authors found that difficulties in untangling causal relationships made it virtually impossible to generate robust counterfactual estimates of programme impacts. Recently, the OECD, the DSP, Ireland and the European Commission’s Joint Research Centre (2024) published a study on two large public works programmes in Ireland, the Community Employment (CE) scheme and Tús, in which the authors evaluated labour market and non-labour market outcomes of participants of these programmes. These programmes provide job placements in local communities to long-term jobseekers and disadvantaged groups; the work is designed to help the community, and it involves community and voluntary organisations. The programmes are administered by the DSP. In particular, CE was designed as an integrated training and employment programme, so primarily as a labour market 23 | Literature scheme and only secondarily as a community development scheme. The training element of the CE is funded by the NTF. The OECD et al. (2024) built a rich dataset by linking the DSP’s data on unemployment, welfare benefit receipt and detailed participation data on Tús and CE to administrative data from the Office of the Revenue Commissioners (Revenue) on earnings and weeks of employment. The dataset contains background information on jobseekers, details on their participation in CE, and individual outcomes after participation. To assess the impact of CE on labour market and non-labour market outcomes of programme’s participants (treatment group), a suitable control group was built of non-participants to CE who were eligible for the scheme and very similar in their observable characteristics to actual CE participants. PSM techniques were used to address selection bias issues. The impact of CE participation on employment and earnings follows a U-shaped pattern, with short negative effect directly after participation (‘lock in’ effects) and positive outcomes in the medium and long-term (after two/three years). When evaluating non-labour-market outcomes, it was found that participation in CE reduces the probability of claiming disability benefits in comparison to the control group. 38 However, the study cannot fully assess the many outcomes CE might offer in terms of ‘social inclusion’ because data on these outcomes are not available. The impact of CE on labour market and social inclusion outcomes is not the same for all types of CE participants, but varies markedly across subgroups of jobseekers (OECD et al., 2024). The report authors highlighted many challenges and difficulties related to the dataset construction process for the study and recommended improving the current situation of data infrastructure in Ireland: Information was frequently available on the same benefit or scheme participation in more than one data source. […] Piecing together these different sources of information was never easy, as very rarely the two sources were providing coherent information. The working solution to this issue was to identify what could be considered the ‘main’ source of information on each scheme or benefit, in terms of coverage and likely reliability of data. This leaves however an open question on comparability between different datasets, reliability of the information provided, and in general, an impression that availability of more comprehensive meta-data would help interpreting the information found when exploring the data sources. As the source datasets are built for operational purposes, developing comprehensive 38 CE participation was also found to enhance take-up of education subsidies, such as the BTEA, in the medium-long term (after three years). While it is positive to see community employment participants choosing to engage in education, the previously outlined findings with regards to the effectiveness of the BTEA scheme need to be borne in mind. Roadmap for an evaluation of the National Training Fund |24 meta-data documentation will assist in re-using them for analytical purposes. Furthermore, the development of a longitudinal database that can consider a variety of data points and use them to come to some determination of labour market status, would alleviate many of the data construction challenges. (OECD et al., p. 132) 3.3.2.3 Firm-level outcomes Very few evaluations have been carried out on the impact of training programmes on firms in Ireland. The few studies that do exist focus on measuring the differential impacts of different forms of training grant assistance on the performance of assisted firms. Roper and Hewitt-Dundas (2001) analysed data on small manufacturing companies in Northern Ireland and the Republic of Ireland from 1991 to 1994; their sample included companies that did not receive any type of grant and companies in receipt of either a workforce training grant or other type of grant, such as for marketing or capital costs. Results showed that firms in assisted clusters had positive results in terms of employment growth over a threeyears horizon, compared to non-assisted firms, but no effect was found regarding profitability or turnover growth. With a similar dataset, McGuinness and Hart (2004) examined the impact of different forms (in nature and timing) of grant assistance to firms in Northern Ireland from 1994 to 1997. They found that grant assistance generally boosts employment growth, especially in smaller firms, while the effects on turnover and productivity vary by type of grant, with marketing grants benefiting larger firms in the medium term. 3.3.3 Summary of NTF-funded initiative evaluations In Table 3.1 below, we summarise recent findings from existing studies regarding evaluation of NTF-funded initiatives. Please note that the table aims to provide a fairly comprehensive overview of these studies but cannot be considered exhaustive. The studies employ various methodologies, which are outlined in Section 3.1.2. 31 | Roadmap for an evaluation of the National Training Fund Hard outcomes such as jobs obtained, numbers of qualifications, and numbers progressing onto further education and training (though useful in some cases), do not show the success of the project as a whole. They are an insufficient indicator of a beneficiary’s increased employability. Target groups that are facing multiple barriers to employment may be a long way from being able to acquire a qualification or employment. Consideration of soft outcomes for such groups is a crucial indicator of success. Measuring soft outcomes can also help with the national level evaluation to provide a fuller picture of the impact of the programme as a whole. (Dewson et al., 2000, p. 4). Such outcomes of interest can be categorised under four headings: key work skills (e.g., teamwork, communication, literacy, timekeeping), attitudinal skills (e.g., motivation, confidence, responsibility, self-esteem), personal skills (e.g., appearance, attendance, timekeeping) and practical skills (e.g., ability to complete forms, manage money, complete a CV). As outlined above, focusing on employment outcomes alone for such a marginalised group may give a misleading picture of programme impact. The Economic and Social Research Institute (ESRI) has recently completed a mixed-methods study of pre-employment supports delivered under Pobal’s Social Inclusion and Community Activation Programme (SICAP) (Whelan et al., 2020). 44 The research combines a counterfactual estimate of immediate employment impacts with case study and survey evidence aimed at identifying the softer impacts of the programme. In the sub-sections below, we discuss: existing datasets that would facilitate CIE evaluation of NTF labour market initiatives; and datasets that present significant data gaps and challenges to CIE evaluation. 4.2.1 Datasets that facilitate CIE evaluation of NTF-funded initiatives As part of the study methodology, we investigated multiple data sources to assess their suitability for evaluating NTF-supported programmes. In this process, we identified several key datasets that can provide valuable insights into learner outcomes, employment transitions and progression. This section discusses these datasets, their potential for CIE, and the key challenges associated with their use. Once NTF-funded programmes are systematically identified, it will be possible to compare NTF beneficiaries and non-NTF beneficiaries using robust econometric methods to assess training impacts on employment, earnings and job quality. Table 4.1 compares six key datasets that have the potential to be used for evaluating NTF labour market initiatives. Each dataset is comprehensive in 44 Pobal. ‘Social Inclusion and Community Activation Programme (SICAP) 2024–2028’. Evaluation possibilities for the National Training Fund |32 capturing core variables such as programme participation, unique identifiers and demographic details (age, gender, education and employment history). Outcome variables are more uneven: while most cover labour market status, only some include earnings, completion or certification information. Programme-level variables, such as type, duration, delivery mode and intensity, are generally well represented in the datasets, though with gaps across each. Additional explanatory variables, including social welfare payments, health, migration status and location, are available in some but not all six key datasets. Overall, the table highlights that while there is a strong foundation for evaluation, no single dataset is comprehensive, making linkage or mixed-use necessary to fully capture programme impacts. 4.2.1.1 The Educational Longitudinal Database The Educational Longitudinal Database (ELD) is, potentially, one of the most valuable resources for evaluating the impact of NTF labour market initiatives, given that it tracks learner outcomes over time, and for a given cohort provides access to a comparable control group. Established by the Central Statistics Office (CSO), since approximately 2010 it has served as a statistical framework for the compilation and analysis of learner outcomes. It provides the basis for a series of projects that the CSO has established in collaboration with Irish public sector bodies to examine learner outcomes across a range of educational levels and programmes, the most recent of which is its Apprenticeships project. 45 By linking data on individuals who have completed training programmes with employment, welfare and further education records, the ELD can potentially enable a comprehensive analysis of the effectiveness of NTF-supported initiatives. The ELD is produced by matching datasets on learners who have completed courses or programmes to other datasets that describe their outcomes in subsequent years. The data sources used to describe learner outcomes include employment and self-employment datasets from Revenue, benefits data from the DSP, and data on educational participation from the Department of Education and several state agencies, including the HEA, Quality and Qualifications Ireland (QQI), and SOLAS. 46 This database allows for the assessment of key employment outcomes, such as job placement rates, earnings growth and job retention, using employment and selfemployment records from Revenue. It also helps measure the extent to which training reduces reliance on social benefits by integrating data from the DSP. Additionally, the ELD facilitates an understanding of lifelong learning pathways by 45 CSO (2021). ‘Key Findings apprenticeship outcomes – Qualification year 2020’. 46 For more information, please see: https://www.cso.ie/en/methods/education/educationallongitudinaldatabase/educationallongitudi naldatabaseeld/. 33 | Roadmap for an evaluation of the National Training Fund incorporating information from the Department of Education, the HEA, QQI and SOLAS. By applying CIE techniques, the ELD could be used to determine whether training programmes funded by the NTF lead to better labour market outcomes, when those of participants are compared to similar individuals who did not participate. Most importantly, a control group can be defined within the ELD dataset, by identifying individuals within the chosen cohort who did not participate in the NTFfunded initiative of interest. This evidence can support data-driven decisions to improve training fund allocation and programme design. 4.2.1.2 Jobseekers Longitudinal Dataset and Work and Welfare Longitudinal Database In evaluating the effectiveness of active labour market policies (ALMPs) in assisting people into employment in Ireland, two significant datasets are utilised: the Jobseekers Longitudinal Dataset and, in more recent times, the Work and Welfare Longitudinal Database (WWLD). The Jobseekers Longitudinal Dataset, which has been replaced by the WWLD, was developed and maintained by the DSP to track social welfare claims, employment, training and activation programme interventions of all individuals who have made a jobseeker or one-parent family payment claim since 2004. This is a comprehensive dataset that has enabled researchers to analyse the trajectories of jobseekers over time, providing valuable insights into the impact of labour market initiatives on individual’s transitions from unemployment to employment. The more recent WWLD has replaced the Jobseekers Longitudinal Dataset, and offers a broader perspective on individuals’ interactions with the welfare system and the labour market. By linking data from various administrative sources, the WWLD facilitates a more detailed analysis of employment patterns and the effectiveness of different ALMP interventions. According to the Organisation for Economic Co-operation and Development’s (OECD) (2024) technical report on the impact evaluation of Ireland’s ALMPs, both these longitudinal datasets are instrumental in assessing programme outcomes and informing policy decisions. However, it should be noted that Ireland’s most recent seasonally adjusted unemployment rate (April 2025) stood at 4.1 per cent , down from a rate of 4.4 per cent in April 2024. This low rate and its steady decline over time indicate that the labour market is approaching full employment. Therefore, in such a context, the relevance of ALMPs may be perceived as diminished, given the reduced pool of unemployed individuals. However, even in a tight labour market, ALMPs play a crucial role in addressing structural unemployment, supporting vulnerable groups and enhancing workforce adaptability to economic shifts. It is also possible, given the open nature of the Irish economy and its susceptibility to external shocks, that unemployment will begin to rise again at some point in the future. Therefore, continuous evaluation using datasets such as the WWLD remains essential to Evaluation possibilities for the National Training Fund |34 ensure labour market policies funded by the NTF effectively meet the evolving needs of the labour market. 4.2.1.3 Facilitating a CIE approach: ELD and WWLD Drawing from our data reviews and interviews with relevant stakeholders, we conclude that the ELD and the WWLD together represent a strong foundation for applying a CIE approach to assess the impact of NTF labour market initiatives in Ireland. However, since the ELD is maintained by the CSO and the WWLD by the DSP, it is essential for these bodies to collaborate with DFHERIS to secure access to anonymised data or relevant data samples necessary for effective programme evaluation. By linking data on learners who have completed training programmes with post-training employment and social welfare records, the ELD can potentially enable a robust analysis of labour market outcomes. As mentioned already, once NTF-funded programmes are systematically identified, either from existing records or through improved tracking mechanisms, the ELD and WWLD can facilitate comparisons between training participants and non-participants, thus allowing for the application of robust econometric techniques, such as difference-indifferences and propensity score matching (PSM). Such methodologies will help establish the causal impact of training interventions on employment, earnings, job quality and occupation related outcomes. 4.2.2 Datasets with significant gaps and challenges to evaluation Several other existing datasets, shown in Table 4.1, contain high-quality data, and contain high coverage of personal public service number data (PPSN), which allows for longitudinal tracking. These include the HEA’s Application Management System (AMS), SOLAS’s Programme and Learner Support System (PLSS) and the NAO’s Apprenticeship Client Services System (ACSS). In fact, some of these high-quality datasets (AMS and PLSS, shown in Table 4.1) serve as foundational sources for the construction of the ELD database. However, used in isolation, there are critical gaps in control group data and in regard to the systematic collection of outcome variables at appropriate time intervals. In order to strengthen CIEs, it is essential to collect information on individuals who apply but do not enrol in training programmes, as these individuals can serve as a natural comparison group. Even if the case where treatment and control firms differ in terms of observable and unobservable characteristics, difference-in-differences and PSM are econometric techniques that can be employed to ensure that any counterfactual estimates of the impact are robust. In the absence of an effective control group, at a minimum, the impact of different forms of grants within assisted firms should be assessed (see McGuinness and Hart, 2004). There is a need for structured follow-up surveys to systematically capture key outcome variables, qualitative insight, and self-reported employment or skills outcomes that administrative data alone may not reveal. Such enhancements 35 | Roadmap for an evaluation of the National Training Fund would improve capacity to assess the impact of NTF-funded programmes over time. Programmes in receipt of European Social Fund (ESF) funding survey participants at three key timepoints: at the start, on completion (after four weeks), and after six months. For unemployed individuals, transitions into employment represents the primary measure, while for those already employed the focus is on self-assessed improvements in labour market status, such as wage increases or promotions. Thus, the evaluation of NTF programmes in Ireland can also be informed by established ESF methodologies and evaluations. The European Commission provides detailed guidance on evaluation design and implementation, including the European Social Fund+ Evaluations resource, which sets out standards for evaluation practice, and the data and evaluation guidelines, which cover outsourcing arrangements, practical challenges involved in assessing programme impacts and the use of administrative data for monitoring and assessment purposes (European Commission, 2021; 2020; 2019). ESF evaluators have highlighted issues with outcome data collected after six months, where selfreported surveys achieve very low response rates and relative sample sizes. 47 This limits reliability and risk bias in the estimation of impacts, even with representative sampling. 48 By contrast, administrative data offer fuller coverage, avoid recall issues and enable more robust long-term tracking. The development of the Government’s Virtual Data Rooms service, which is under the remit of the CSO, may facilitate increased government department and state agency data sharing, linking and availability. The purpose of this service is to allow public sector organisations to match their own data with additional pseudonymised data provided by other public sector organisations, with the intention of supporting evidence-based policy making, and policy evaluation and formation. Thus, this service could enhance future evaluations of the NTF by providing more streamlined access to verified personal/firm-level information. 49 However, at present the planned rollout of this service has only begun (Quarter 2 of 2025). Thus, it remains unclear how this service will function in practice, particularly in relation to administrative datasets and research access. While the CSO have robust data governance arrangements in place, and apply strict terms and conditions for accessing data, due to the CSO being governed by the 1993 Statistics Act, the potential application of this service for evaluation and policy 47 These issues might necessitate adjustments in the survey methodology, including enhanced engagement strategies, improved tracking systems or alternative data collection methods to ensure that sufficient and reliable follow-up information is gathered. 48 Examples of effective data collection, sharing and evaluation practices have been demonstrated for the ESF in Germany, Holland and Sweden (information provided by DFHERIS during meeting on 10 February 2025, and subsequent email follow-up on 20 May 2025). 49 The establishment of this service formed part of the Government’s 2019–2023 Public Service Data Strategy: public-service-data-strategy-2019-2023-ae1c4cdf-b7e4-4e0c-bca7-b84b962a8ad3.pdf. The service’s pilot was completed in quarter one of 2025. For more information on this service, please see: VDR_Update_27_June_2024_for_ESLG.pptx. Evaluation possibilities for the National Training Fund |36 analysis will depend on how interoperability and data-sharing frameworks are implemented. The School Leavers Survey was a long-running study that examined young people’s experiences in school and their transition to the labour market, further education or economic inactivity. 50 The Irish Social Science Data Archive holds data for this survey from 1980 to 2007, after which the survey was discontinued. If reinstated, this survey could provide critical control group data, allowing for more accurate CIEs of training programmes. By capturing information on individuals who do not engage in NTF-funded training programmes, having access to such a dataset could enhance the ability to compare outcomes between participants and nonparticipants, strengthening the overall accuracy of NTF evaluations. Failing these additions, there remains the possibility to comparing the outcomes across various NTF-funded training programmes relative to one another, where the necessary data exists, and there is no access to a comparable control group. (See McGuinness and Harte (2004) for an example of an approach that compares the impacts of interventions across treatment groups.) As a recent example of data gaps, McGuinness et al. (2018), in their evaluation of the Post-Leaving Certificate (PLC) programme for SOLAS, identified significant data limitations, particularly in relation to the absence of a control group, which hinders the establishment of a counterfactual scenario. This limitation restricted capacity to assess the direct impact of PLC programmes on participants’ outcomes. In order to address this challenge, the ESRI, in conjunction with SOLAS, employed a surveybased methodology. Surveys were designed and administered, targeting both PLC graduates and a comparison group of Leaving Certificate graduates who did not enrol in PLC courses. This approach facilitated the collection of data on both educational and employment trajectories, enabling a comparative analysis between the two groups. By analysing the responses, McGuinness et al. (2018) estimated the counterfactual – i.e., what the outcomes for PLC participants might have been had they not enrolled in the programme. This allowed for an assessment of the PLC programmes’ effectiveness in enhancing educational and employment outcomes. Participation in PLC programmes was found to have positive impacts on both educational progression and employment outcomes. Specifically, PLC graduates were more likely to continue to higher education compared to similar Leaving Certificate graduates, and they also were found to experience improved employment prospects. It is important to note that this survey-based approach with primary data collection, while useful in the absence of a control group, is often more expensive and time-consuming than using linked administrative data. Administrative data, if available, allows for more cost-effective and comprehensive analysis by 50 The School Leavers Survey was conducted by the ESRI and funded by the Department of Education. 37 | Roadmap for an evaluation of the National Training Fund leveraging existing records on employment, education and welfare interactions. It also reduces complexities such as survey non-response and recall bias, thereby improving the reliability of impact evaluations. As such, the re-establishment of some form of school leavers survey would enhance capacity to evaluate NTFfunded labour market initiatives, including any current or future NTF initiatives targeted at younger cohorts. Another key consideration to be aware of is that the timing of evaluations is a crucial factor in measuring shortto long-term impacts. A phased approach to evaluation should be considered; one might take place, for example, 6–12 months post-training to assess immediate employment outcomes, job retention and initial earnings changes. After two to three years post-training, it could evaluate mediumterm effects, such as career and earnings progression. Some consideration of the overall macroeconomic climate or other events, such as the COVID-19 pandemic, are necessary to ensure that NTF-funded programme participants have had the necessary time to embed the skills developed into their careers. Longer term labour market impacts, such as wage growth and career stability, can be analysed 5–10 years post-training. Lock-in effects occur when participants in education or ALMPs temporarily reduce their job search efforts during programme participation, leading to short-term decreases in employment rates. Such lock-in effects need to be taken into consideration when deciding on the timing of evaluations. Additionally, for certain highly specialised or upskilling programmes, longitudinal tracking over a decade may be necessary to fully understand their impact on career trajectories and earnings mobility. For instance, only short-term post-training information is collected in the case of Springboard+ (HEA dataset). Furthermore, in an evaluation, outcomes might also relate to the potential differential effects of mode of delivery; i.e., online, in-person or blended teaching and learning. These can vary by, for example, intensity or duration of courses. Another important issue to mention is that available datasets through Revenue capture total weekly or monthly pay, but lack data on the number of hours worked. This gap creates problematic measurement errors, particularly when evaluating the earnings impact of training programmes, as an increase in total earnings could reflect more hours worked rather than higher wages. To enhance wage-related analysis, it is essential to incorporate hourly wage estimates, either by encouraging the collection of standardised work-hour information within administrative datasets or through self-reported survey data. This would allow for more precise assessments of whether training improves wage rates, not just other employment conditions (for example, a move from part-time to full-time or increased hours worked). In this regard, towards the end of 2024, the Department of Enterprise, Trade and Employment (DETE) submitted a business case to Revenue for the collection of hours worked data through its PAYE Modernisation (PMOD) Evaluation possibilities for the National Training Fund |38 system. 51 The business case was led by DETE, in collaboration with the CSO and other interested policy bodies supporting the proposal. 52 The DETE proposal also has strong support from the National Statistics Board (NSB) (NSB, 2024). Effective evaluation requires technical expertise in econometrics, statistical analysis and labour economics. It is necessary for evaluators to have proficiency in advanced quantitative methods and econometrics, including the ability to construct counterfactual groups, assess selection bias and interpret causal effects. To ensure objectivity, it is also crucial that evaluations are conducted independently, whether through external research institutions/bodies, universities or dedicated evaluation units within government agencies. Independence facilitates the prevention of any conflicts of interest and ensures increased credibility in the findings for policymaking. Mixed-methods approaches to evaluation integrate both quantitative and qualitative research methods, offering a comprehensive understanding of complex programmes or interventions. By combining statistical data with detailed personal insights, these approaches provide a more nuanced analysis than singular methods. For instance, as mentioned above, Whelan (2020) evaluates preemployment supports under Pobal’s SICAP programme, employing a mixedmethods design. Quantitative data were used to assess employment outcomes and qualitative interviews were conducted with participants and staff to improve understanding of personal experiences and challenges. This combination allows for a more holistic approach to evaluation, capturing both measurable impacts and the contextual factors influencing those outcomes. Such approaches are particularly beneficial when exploring more complex social programmes where statistical data alone may not capture the full scope of participant experiences or programme effectiveness. By integrating diverse data sources, mixed-methods evaluations can validate findings across methodologies, enhance the reliability of results and provide actionable insights for policymakers and practitioners. Mixedmethods evaluations are markedly more expensive than purely quantitative approaches due to the additional costs of conducting qualitative research, such as interviews, focus groups and necessary fieldwork. These methods require more time, researchers with these specific skills and resources for data collection and analysis, making them more resource-intensive compared to relying solely on existing administrative or survey data as suggested above. Nevertheless, mixedmethod approaches are key when it comes to the evaluation of complex labour market, or any other, initiatives. 51 PMOD, introduced by Revenue in 2019, is a system that requires employers to report employee pay in real time for each payroll cycle. It replaced the practice of submitting an annual P35 form. 52 DSP, the Department of Children, Equality, Disability, Integration and Youth and the Low Pay Commission. 39 | Roadmap for an evaluation of the National Training Fund TABLE 4.1 LABOUR MARKET INITIATIVES – DATA INFRASTRUCTURE Application Management System (AMS) – HEA Apprenticeship Client Services System (ACSS) – NAO Educational Longitudinal Database (ELD) – CSO Programme and Learner Support Systems (PLSS) – SOLAS Skillnet Ireland data Work and Welfare Longitudinal Dataset (WWLD) – previously Jobseeker Longitudinal Dataset – DSP Core variables for evaluation - NTF programme participation indicator(1) ✓(8) ✓ ✓ ✓ ✓ ✓ - Unique identifier/ PPS number ✓ ✓ ✓ ✓ ✓ - Company registration number (9) ✓ Outcome variables (before and after) Individual-level: - Completion ✓ ✓ ✓ (11) - Labour market status (e.g., employment, unemployment, additional education/training) ✓ ✓ ✓ ✓ ✓ ✓ - Earnings/wages ✓ ✓ - Career progression (e.g., occupation upgrading, hours worked, moving from part-time to full-time) - Certification ✓ ✓ Programme information: - Type of programme ✓ ✓ ✓ ✓ - Duration of training (hours/days/months/years) ✓ (2) ✓ ✓ Evaluation possibilities for the National Training Fund |40 TABLE 4.1 (CONTD.) LABOUR MARKET INITIATIVES – DATA INFRASTRUCTURE Application Management System (AMS) – HEA Apprenticeship Client Services System (ACSS) – NAO Educational Longitudinal Database (ELD) – CSO Programme and Learner Support Systems (PLSS) – SOLAS Skillnet Ireland data Work and Welfare Longitudinal Dataset (WWLD) – previously Jobseeker Longitudinal Dataset – DSP - Mode of delivery (in-person, online, blended) ✓ ✓ ✓ ✓ ✓ ✓ - Intensity (i.e., full-time/part-time) ✓ ✓ ✓ ✓ - Start date ✓ (2) ✓ ✓ ✓ - Finish date ✓ (2) ✓ ✓ ✓ Explanatory variables Individual-level: Core variables - Educational attainment ✓ ✓ ✓ ✓ ✓ ✓ - Age ✓ ✓ ✓ ✓ ✓ ✓ - Gender ✓ ✓ ✓ ✓ ✓ ✓ - Employment history (e.g., employed in last month, in last year) * ✓ ✓ ✓ ✓ ✓ ✓ - Migrant/non-Irish ✓ (12) ✓ ✓ Secondary variables - Social welfare payment (Jobseeker’s Allowance) ✓ ✓ ✓ ✓ - Own transport (e.g., car) ✓ - Long-term unemployment (12 months or more) ✓ ✓ ✓ ✓ ✓ - Health (3) ✓ - Public sector employment scheme (Community employment scheme) ✓ 47 | Roadmap for an evaluation of the National Training Fund TABLE 4.2 FIRM-LEVEL INITIATIVES – DATA INFRASTRUCTURE Variables IDA data Engineers Ireland data Skillnet Ireland data Training Grant to Industry – Enterprise Ireland Core variables for evaluation - NTF programme participation indicator ✓ ✓ ✓ ✓ - Unique identifier/ PPS number - Company registration number ✓ Outcome variables (before and after) Firm-level: - Firm turnover ✓ - Employment numbers ✓ (only before) ✓ ✓ (only before) ✓ - Labour productivity - Sales volume ✓ (only before) ✓ Programme information - Type of programme ✓ ✓ - Duration of training ✓✓ ✓ - Mode of delivery (in-person, online, blended) ✓ ✓ - Intensity (i.e., full-time/part-time) ✓ ✓ ✓ - Start date ✓ ✓ ✓ - Finish date ✓ ✓ ✓ Explanatory variables Individual-level: Evaluation possibilities for the National Training Fund |48 TABLE 4.2 (CONTD.) FIRM-LEVEL INITIATIVES – DATA INFRASTRUCTURE IDA data Engineers Ireland data Skillnet Ireland data Training Grant to Industry – Enterprise Ireland - Employment history (e.g., employed in last month, in last year, etc.) ✓ ✓ - Educational attainment ✓ ✓ - Age ✓ ✓ - Gender ✓ ✓ ✓ - Migrant/non-Irish - Additional professional/vocational training or skills - Occupation ✓ ✓ - Industry/sector ✓ ✓ Firm-level: - Firm size (number of employees) ✓ ✓ ✓ ✓ - Ownership type (e.g., multinational) ✓ ✓ ✓ ✓ - Sector/industry ✓ ✓ ✓ ✓ - Geographic location type (e.g., geocode, county, rural/urban) ✓ ✓ ✓ ✓ - Company age Note: Individual-level outcome variables could also be collected (i.e., training completion, labour market status, earnings/wages, career progression, certification). Some secondary individual-level dependent variables could be collected (as per labour market initiatives), but this was not strictly necessary. Source: Information obtained from grantee organisations. 49 | Roadmap for an evaluation of the National Training Fund 4.4 SOCIAL AND OTHER INITIATIVES Within the academic and policy literature, there exists a number of theoretical frameworks that could potentially be used for measuring social and community level initiatives. Examples include: the Logic model (Milstein and Chapel, 2011); the four pillars approach (Pritchard and Kazimirski, 2014); the ABCD framework (Barr and Hashagen, 2000); and the LEAP model (Barr and Dailly, 2007). All of these provide suggestions of how progress at the community level can be measured. While it is not clear that any particular one of these conceptual frameworks should guide the measurement of NTF programmes with a social or other focus, each of these theories generally involves a clear statement on programme objectives, which are linked explicitly to inputs, processes and outcome variables that the policy should be influencing. As discussed above, counterfactual analysis allows the outcomes of the intervention to be compared with the outcomes that would have been achieved in the absence of the intervention. International best practice for evaluating community and social level programmes was extensively assessed in Whelan et al. (2019). This study specifically looked at SICAP in Ireland and examined how programme impacts could be effectively measured. After evaluating the international literature, a principal conclusion of the study was that difficulties in untangling causal relationships made it, in this instance, virtually impossible to generate robust counterfactual estimates of programme impacts. The analysis pointed out that it was impossible to identify causal links between community level expenditure and general levels of community wellbeing metrics, such as those included in the Community Tool Box developed by the University of Kansas Work Group for Community Health and Development. 60 Proposed metrics suggested in the Community Tool Box attempt to measure community-level well-being using very broad aggregates that are not linked to any particular policy intervention (Milstein and Chapel, 2011). Examples of such metrics include measures of: income, poverty, deprivation, educational attainment, unemployment rates, workforce entry, social welfare payment discontinuation, community participation, membership in clubs and community associations, number of community activists, citizen advocacy groups and organisations, political participation (percentage of individuals voting), diversity of population, average price of a single family house, average rental rates, average commuting times, number of (current and new) local businesses, local revenue form taxes and fees, number of service firms, number of new commercial buildings being constructed and occupancy rates. 60 For more information, please see https://ctb.ku.edu/en. Evaluation possibilities for the National Training Fund |50 Across the literature, we have found no evidence of any systematic attempts to practically measure a counterfactual estimate of community level outcomes. Our previous research has highlighted many reasons why this might be the case. Specifically, Whelan et al. (2019) point out that many factors will simultaneously affect these outcomes, and that attempting to disentangle and isolate the impacts of any individual policy intervention is extremely difficult. In these situations, Whelan et al. (2019) propose the adoption of a monitoring framework that focuses on metrics specific to the objectives of the programme. However, even when programme-specific metrics are established, causal relationships are extremely difficult to extract given the structure of funding to bodies implementing community level programmes. 61 Other barriers to measuring counterfactual impacts at a community or social level include the difficulty involved in identifying: control groups where no community or social assistance took place; appropriate common outcome metrics given the diverse objectives of community and social groups; and the appropriate timeframe over which impacts should be measured. The principal barriers to identifying causal outcomes for community level expenditure are summarised in Figure 4.1. The key challenge is that existing data may not allow a clear demonstration of a causal link between a policy intervention and changes in broad measures of community wellbeing, shown by the break in the circular overview (Figure 4.1) between points (1) and (3). 61 For example, in 2016, the average SICAP funding was found to account for an average of approximately 16 per cent of the total budgets to programme implementers (Darmody and Smyth, 2018). 51 | Roadmap for an evaluation of the National Training Fund FIGURE 4.1 CIRCULAR OVERVIEW OF COMMUNITY LEVEL POLICY IMPACT EVALUATION Source: Whelan et al. (2019). Whelan et al. (2019) identify the following confounding factors contributing to the difficulties involved in trying to estimate the causal impact of policies targeting community well-being: • Numerous national agencies simultaneously implement policies that will affect such broad outcomes, making it difficult to isolate the impacts of one particular policy. • Local organisations targeting specific communities often receive funding from multiple sources, making it impossible to measure the impact of a particular funding stream, even in instances where the community level outcome measures are narrowly defined and identifiable. • It may be more feasible to focus on more narrow outcomes for the purpose of evaluating the impact of funding to community level organisations. However, local community organisations tend to be highly heterogeneous in nature with differing objectives, making it extremely difficult to identify a set of specific community level outcome measures relevant to the activities of all funded groups. • It is extremely difficult to identify control groups at a community level who have not been subject to any policy interventions against which to measure the counterfactual impact of an intervention. Therefore, the expectation of measuring any causal influence of SICAP on broad community level outcomes, such as poverty rates or levels of educational attainment, was not felt to be practical in Whelan et al. (2019). This was due to the Evaluation possibilities for the National Training Fund |52 existence of various streams of funding targeting such outcomes and the overall complexity of the system. Nevertheless, community and social development initiatives, as with all government-funded activities, require monitoring and measurement. Whelan et al. (2019) suggest that the most appropriate framework for assessing the impact of community level expenditure involves approaches such as a logic model framework linked specifically to programme objectives, which would allow for the monitoring of key outcome variables over time. A number of other similar monitoring frameworks could be considered, including the four pillar approach, the ABCD model and the LEAP framework. Further recommendations include the adoption of a community level ‘distance travelled tool’ and/or the commissioning of thematic qualitative studies that periodically collect evidence of themes related to programme goals. In the sub-section below, Section 4.4.1, we discuss the NTFfunded ‘social and other’ initiatives where the data available are more suited to monitoring than a CIE approach. 4.4.1 Suitable for monitoring Social and community level initiatives that are funded by the NTF are not suitable for standard CIE approaches. The Wheel, a national association of charities, community groups and social enterprises, receives funding from the NTF alongside other sources. Through its NTF-funded initiative, it is providing leadership, upskilling, reskilling and other education opportunities to people working or volunteering in the charity and community sectors in Ireland. Thus, the initiative’s focus is on workforce development, and it seeks to support individuals concerned in their community development, social and charity-related work. As a result, it is more appropriate to evaluate this and similar programmes, as outlined above, through systematic monitoring, qualitative case studies and thematic reviews, rather than using CIE methods. Developing performance indicators tailored to these interventions, such as skill acquisition in informal settings, can provide valuable insights into the effectiveness of the programme. 4.5 NTF EVALUATION FEASIBILITY ANALYSIS In this section, we aim to provide a structured overview of the evaluation possibilities for the NTF. Table 4.3 categorises each NTF-funded initiative with the associated grantee organisation and allocated funding (for 2023) under the type of programme concerned – labour market initiative, firm-level initiative, social/other initiative or cross-cutting initiative. Furthermore, it classifies each area of funding according to the following categories: • complete CIE (fully); • partial evaluation that can identify differential impacts (partially); • only minor adjustments are needed to enable evaluation (easily resolved); • monitoring is the most appropriate approach (monitor); • no evaluation is possible currently (not possible). 53 | Roadmap for an evaluation of the National Training Fund As can be seen from Table 4.3, and in line with our discussions in the sections above, currently labour market initiatives (as opposed to firm-level ones) are best positioned for robust CIE. This is due to the greater availability of relevant datasets and established outcome variables for these initiatives. In contrast, firm-level initiatives currently face more significant challenges that impede effective CIEs. These initiatives require considerable changes in their data collection (e.g., comprehensive collection of company registration numbers) and measurement of outcomes to facilitate more accurate and comprehensive assessments. Addressing these issues is critical for ensuring that firm-level initiatives can be evaluated similarly to their labour market counterparts in the future. Overall, approximately 71 per cent of NTF funding is allocated to programmes that appear to support a complete CIE; approximately 1 per cent corresponds to initiatives suitable for partial evaluation to identify differential impacts. A further 6.7 per cent of NTF funding is associated with initiatives requiring changes that should be easily resolved to enable evaluation. Monitoring is found to be the most appropriate approach for initiatives claiming approximately 6 per cent of NTF funding. Finally, 16 per cent is associated with programmes for which no evaluation is currently possible. Evaluation possibilities for the National Training Fund |54 TABLE 4.3 NTF EVALUATION POSSIBILITIES (PRELIMINARY FINDINGS) Programme Type Grantee organisation Fully Partially Easily resolved Monitor Not possible Funding 2023 Share of total NTF funding in 2023 (€000) (%) LM/ F/O/CC Counterfactual Differential impacts Changes required Monitoring more suitable No evaluation possible at present Training People for Employment LM SOLAS 287,095 31.6 Apprenticeship1** LM/CC SOLAS/NAO 195,936 21.6 Enterprise-Focused Higher Education Provision1 LM/CC HEA 148,352 16.3 Apprenticeship1*** LM/CC HEA/NAO 78,940 8.7 Human Capital Initiative (Pillars I & II)1 LM/CC HEA 6,988 0.8 Springboard+: In Employment LM HEA 29,550 3.3 Springboard+: For Employment LM HEA 6,888 0.8 Employee and Continuing Professional Development LM SOLAS 22,783 2.5 Community Employment Training2 LM/CC DSP 4,023 0.4 Training Networks Programme – Seeking Employment LM Skillnet Ireland 5,449 0.6 Training Support Grants LM DSP 2,634 0.3 SOLAS Traineeship1 LM/CC SOLAS 2,900 0.3 Work Placement Experience Programme LM DSP 835 0.1 55 | Roadmap for an evaluation of the National Training Fund TABLE 4.3 (CONTD.) NTF EVALUATION POSSIBILITIES (PRELIMINARY FINDINGS) Programme Type Grantee organisation Fully Partially Easily resolved Monitor Not possible Funding 2023 Share of total NTF funding in 2023 (€000) (%) LM/ F/O/CC Counterfactual Differential impacts Changes required Monitoring more suitable No evaluation possible at present Training Networks Programme – Employment1 F/CC Skillnet Ireland 55,692 6.1 Training Grants to Industry1,3 F/CC Enterprise Ireland 3,500 0.4 Training Grants to Industry1 F/CC IDA Ireland 3,000 0.3 Employee and CPD1 F/CC Engineers Ireland 400 0.04 Human Capital Initiative (Pillar III) S HEA 50,012 5.5 Community and Voluntary Organisations S The Wheel 1,140 0.1 TOTAL NTF FUNDING (€,000) AND SHARE (%) 2023 638,572 (70.5) 6,500 (0.7) 61,141 (6.7) 51,152 (5.6) 148,752 (16.4) 906,117 100.0 Source: Constructed by the authors using data from the Comptroller and Auditor General’s 2022 report on the accounts of the public service and 2023 NTF spending data provided by DFHERIS. Notes: LM – Labour market initiative; F – Firm-level initiative; S – Social/Other initiative; CC – Cross-cutting initiative. 1 Cross-cutting labour market/firm-level initiatives (i.e., initiatives that have firm-level spillover effects); and 2 cross-cutting labour market/social/other initiatives (i.e., initiatives that have social/other spillover effects). 3 Enterprise Ireland’s ‘training grants to industry’ consist of two main programme categories: ‘Leadership, management development & scaling supports’ (which consists of nine individual long and short duration programmes) and ‘Mentor network’. Presently, both programme categories would appear to be evaluable using a differential impacts methodology. 4 The total NTF budget allocation for 2023 was €909.437 million, with the remaining €3.320 million of the budget mainly allocated to programmes that research skill requirements for the economy (e.g., the Skills and Labour Market Unit (SOLAS), the Expert Group on Future Skills Needs (DETE), and Regional Skills Fora (SPEE)). * SOLAS’s apprenticeship funding for 2023 includes the NAO’s allocation: since 2023, the NAO has been receiving a separate allocation under the NTF to assist both SOLAS and the HEA in the running of their apprenticeship initiatives, and its allocation for 2023 (€3.400 million) is combined with SOLAS’s apprenticeship allocation (€192.536 million) as NAO’s funding is distributed through SOLAS. 62 ** The HEA also received temporary funding of approximately €4 million for its apprenticeship programme in 2023. 62 Information, including the treatment of the NAO NTF budget in 2023, provided by DFHERIS. Evaluation possibilities for the National Training Fund |56 4.6 SUMMARY In summary, evaluation of NTF-funded initiatives requires a combination of robust data sources, steadfast data confidentiality and data-sharing agreements, welldesigned counterfactual approaches, strategic prioritisation of evaluation efforts, and good collaboration between the Department of Further and Higher Education, Research, Innovation and Science (DFHERIS), the NTF grantee organisations and any external agencies that hold data required to evaluate the NTF (e.g., the CSO). In addition, to allow DFHERIS, and other relevant stakeholders, to make datadriven decisions to optimise the effectiveness of NTF investments in employee and workforce development, it will be important that evaluations of the initiatives are timely, independent and methodologically sound. The ELD and WWLD, supplemented with additional survey data and enhanced administrative data tracking, make for a strong foundation for impact evaluation of NTF-funded labour market initiatives. Since the ELD is maintained by the CSO and the WWLD by the DSP, it is essential for these agencies to collaborate with DFHERIS to enable that Department to secure access to anonymised data, or relevant data samples, necessary for effective programme evaluation, while, at all times, adhering to any data protection legislation (e.g., GDPR and The Statistics Act, 1993). Subsequently, addressing key gaps, such as the comprehensive collection of company registration numbers and control group data, systematic measurement of relevant outcome variables and improved wage data are essential to strengthening the accuracy of CIEs of the NTF funded firm-level initiatives. Given the diverse range of training programmes funded by the NTF, prioritisation will be necessary to determine which programmes should and could be evaluated most readily. The selection process will need to consider factors such as programme cost, scale, alignment with labour market needs and policy relevance. Programmes with a clear skills development component and a direct link to employment outcomes should be prioritised for rigorous CIE. Currently, as mentioned already, the labour market initiatives are best positioned for robust CIE due to the greater availability of relevant datasets and established outcome variables. In contrast, at present, the firm-level initiatives face more significant challenges that impede effective evaluation. As outlined above, these programmes require considerable changes in their data collection and measurement of outcomes to facilitate more accurate and comprehensive assessments. Overall, when we examined current feasibility for evaluation more closely, we found a wide range, with approximately 71 per cent of funding being allocated to programmes that appear to support a complete CIE. In contrast, 16 per cent corresponds to instances where no CIE is possible at present. The remaining 13 per 63 | References Kluve J. (2010). ‘The effectiveness of European active labor market programs’, Labour Econ, Vol. 17, No. 6, pp. 904–918. Kluve, J. (2006). The effectiveness of European active labor market policy, RWI Discussion Papers, No. 37, Rheinisch-Westfälisches Institut für Wirtschaftsforschung (RWI), Essen. Kuku, O., P. Orazem, S. Rojid and M. Vodopivec (2015). Training funds and the incidence of training: The case of Mauritius, IZA Discussion Paper No. 8775, Bonn: Institute for the Study of Labour (IZA). Lalive, R., J.C. van Ours and J. Zweimüller (2008). ‘The impact of active labour market programmes on the duration of unemployment in Switzerland’, The Economic Journal, Vol. 118, No. 525, pp. 235–257. Lenihan, H., M. Hart and S. Roper (2005). ‘Developing an evaluative framework for industrial policy in Ireland: Fulfilling the audit trail or an aid to policy development’, Quarterly Economic Commentary, pp. 69–86. Milstein, B. and T. Chapel (2011). ‘The Community Tool Box: Developing a logic model or theory of change’, Lawrence, KS: University of Kansas, http://ctb.ku.edu/en/tablecontents/sub_section_examples_1877.aspx. McGuinness, S. and M. Hart (2004). ‘Mixing the grant cocktail: Towards an understanding of the outcomes of financial support to small firms’, Environment and Planning C: Government and Policy, Vol. 22, No. 6, pp. 841–857. McGuinness, S., P.J. O’Connell and E. Kelly (2014). ‘The impact of training programme type and duration on the employment chances of the unemployed in Ireland’, Economic and Social Review, Vol. 45, No. 3, pp. 425–450. McGuinness, S., P.J. O’Connell, E. Kelly and J. Walsh (2011). Activation in Ireland: An evaluation of the National Employment Action Plan, ESRI Research Series No. 20, Dublin: ESRI, https://www.esri.ie/publications/activation-in-ireland-an-evaluationof-the-national-employment-action-plan. McGuinness, S., P.J. O’Connell and E. Kelly (2014). ‘The impact of training programme type and duration on the employment chances of the unemployed in Ireland’, The Economic and Social Review, Vol. 45, No. 3, pp. 425–450. McGuinness, S., A. Bergin, E. Kelly, S. McCoy, E. Smyth, D. Watson and A. Whelan (2018). Evaluation of PLC programme provision, Research Series 61, Dublin: ESRI. National Statistics Board (2024). Mid-term review of quality information for all – Numbers matter, Dublin: National Statistics Board. O’Connell, P.J., S. McGuinness, E. Kelly and J. Walsh (2009). National profiling of the unemployed in Ireland, ESRI Research Series Number 10, Dublin: ESRI. OECD (2023). OECD Skills Strategy Ireland: Assessment and recommendations, OECD Skills Studies, Paris: OECD Publishing, https://doi.org/10.1787/d7b8b40b-en. OECD (2019). ‘Getting skills right: Future-ready adult learning systems’, Chapter 5, Getting skills right, https://www.oecd.org/en/publications.html. OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre (2024). Impact evaluation of Ireland’s active labour market policies, connecting people with jobs, Paris: OECD Publishing, https://doi.org/10.1787/ec67dff2-en. Perceptive Insights (2017). 2016 Follow up survey of FET programme participants, report prepared for SOLAS. Roadmap for an evaluation of the National Training Fund |64 Pritchard, D. and A. Kazimirski (2014). Building your measurement framework: NPC’s four pillar approach, London: New Philanthropy Capital, www.thinknpc.org/wpcontent/uploads/2015/04/NPCs-four-pillars-summary.pdf. Roper, S. And N. Hewitt-Dundas (2001). ‘Grant assistance and small firm development in Northern Ireland and the Republic of Ireland’, Scottish Journal of Political Economy, Vol. 48, No. 1, pp. 99–117. Storey, D.J. (2000). Six steps to heaven. Handbook of entrepreneurship, Oxford: Blackwells. Storey, D.J. (2003). ‘Entrepreneurship, small and medium sized enterprises and public policies’, in Handbook of entrepreneurship research, pp. 473–511, Boston MA: Springer. UNESCO (2022). Global review of training funds: Spotlight on levy-schemes in 75 countries, UNESCO. Whelan, A., S. McGuinness and A. Barrett (2021). Review of international approaches to evaluating rural and community development investment and supports, Research Series No. 124, Dublin: ESRI. Whelan, A., S. McGuinness, and J. Delaney (2019). Valuing community development through the Social Inclusion Programme (SICAP) 2015–2017: Towards a framework for evaluation, Research Series No. 77, Dublin: ESRI. Whelan, A., J. Delaney, S. McGuinness and E. Smyth (2020). Evaluation of SICAP preemployment supports, Dublin: ESRI. World Bank (2017). Study on the effectiveness of the Human Resources Development Fund, Washington: World Bank. Wunsch, Conny (2016). How to minimize lock-in effects of programs for unemployed workers, IZA World of Labor, Bonn: Institute for the Study of Labor (IZA), https://doi.org/10.15185/izawol.288. 65 | Appendices APPENDIX I Information gathering template issued to NTF grantee organisations List of questions: 1. What year was the programme established? 2. Please list the groups targeted by the programme. (E.g., employees, unemployed, young people, NEETS, individuals with disabilities, other disadvantaged/marginalised groups, older people, etc.) 3. Is the programme NTF funded only, or part-funded by other agencies/departments? 4. If part-funded, please provide details. (E.g., other agencies/departments involved, the proportion of the programme that is funded by NTF and the proportion funded by other agencies/departments, etc.) 5. Who provides the training/education element of the programme? 6. What body (bodies) has (have) overall responsibility for managing the programme? 7. What is the usual duration of the programme (or the courses covered by the programme)? 8. Describe the nature of the programme. (E.g., pre-employment training, continuing training, in-firm training, business development services to enterprises, etc.) 9. Does the programme target any economic sectors identified as national priorities by the Government, and if so what are the sectors? 10. What are the key objectives of the programme for learners? (E.g., employment, progression to further learning, addressing basic competency gaps, etc.) 11. What are the key objectives of the programme for the firms receiving the training? (E.g., address skill needs/gaps, increasing skill levels of its workforce, increasing workers’ productivity, etc.) 12. Are there any wider objectives of the programme? (E.g., reduction in unemployment, supporting national skills objectives, meeting national skills shortages, increasing productivity, tackling social exclusion etc.) If so, please list. Roadmap for an evaluation of the National Training Fund |66 13. How is the programme (or the courses covered by the programme) delivered? (E.g., classroom, online, work placement, etc.) If the programme is (or the programme courses are) delivered across a range of modes, please provide approximate percentage breakdowns for each mode of delivery. 14. Is the programme (or the courses covered by the programme) certified? If yes, please provide details of the accreditation level and awarding body. 15. Since its establishment, how many people (learners) have commenced the programme (the programme’s courses) on an annual basis? 16. Of those that commenced the programme (the programme’s courses) on an annual basis, what number/percentage completed the programme (their course)? 17. Since its establishment, how many firms have commenced the programme (the programme’s courses) on an annual basis? 18. Of those firms that commenced the programme (the programme’s courses) on an annual basis, what number/percentage completed the programme (the course)? 19. What background data are captured on the programme (course) participants? (E.g., gender, age, educational attainment, previous employment/unemployment history, etc.) 20. At what stage of the programme (course) is this background information captured on participants? (E.g., before commencing the programme/course, weekly, monthly, annually, at programme completion, etc.) 21. What information is captured on those firms that participate in the programme? (E.g., economic sector, number of employees, turnover, multinational or indigenous company, geographic location, etc.) 22. At what stage of the programme (course) is this background information captured on firms? (E.g., before commencing the programme/course, weekly, monthly, annually, at programme completion, etc.) 23. What post-course completion information is captured on programme (course) participants? (E.g., employment, unemployment, further training, emigrated, sector of employment, earnings, etc.) 24. What post-course completion information is captured on firms participating in the programme (course)? (E.g., number of employees, turnover, etc.) 67 | Appendices 25. Is the information that is captured on programme (course) participants linked to any other administrative datasets? (E.g., Revenue data, Department of Social Protection data, etc.) 26. Is the information that is captured on programme (course) learners and firms linked to any other administrative datasets? (E.g., Revenue data, Department of Social Protection data, etc.) 27. Is the programme/course participant PPS number captured? 28. Is the central business register (CBR) number of those firms participating in the training programme captured? 29. How frequently is the post-course completion information on participants captured? (E.g., at the course completion time point, 3 months after, 6 months after, 12 months after, 24 months after, etc.) 30. How frequently is the post-course completion information on firms participating in the programme/course captured? (E.g., at the course completion time point, 3 months after, 6 months after, 12 months after, 24 months after, etc.) 31. Are data captured on those that applied for the programme (course) but were not successful in their application? 32. If yes, please provide details on the information captured on non-participants (background, economic status), along with the frequency at which such information is captured. 33. Are data captured on those firms that applied for the programme (course) but were not successful in their application? 34. If yes, please provide details on the information captured on non-participating firms (economic sector, number of employees, etc.), along with the frequency at which such information is captured. 35. What are the existing key performance indicators (KPIs) for the programme (the programme's courses)? 36. Please provide frequency of collection of KPI data. 37. Has the effectiveness of the programme ever been formally evaluated by an external agency? Roadmap for an evaluation of the National Training Fund |68 38. If yes, please provide details on the evaluation (e.g., methodology employed, data used, who conducted the evaluation, etc.), including a link to/copy of the study. 39. Are programme (course) participants issued with a satisfaction survey on completion of the programme (their course)? If yes, please provide details. 40. Are employers/firms issued with a satisfaction survey on completion of the training programme (course)? If yes, please provide details. 41. Is there ever any qualitative information collected from programme (course) participants, such as through focus groups, workshops, etc.? If yes, please provide details. 42. Is there ever any qualitative information collected from employers/firms, such as through focus groups, workshops, etc.? If yes, please provide details. 43. Please provide links (or references) to any documentation detailing the programme. 44. Please provide links (or references) to any known assessments or evaluations of the programme (publicly available or not). 45. Please provide any other information that you think is relevant for the study for which the information in this template is being sought. 69 | Appendices APPENDIX II Overview of NTF-funded initiatives LABOUR MARKET INITIATIVES APPENDIX TABLE 2.1 APPRENTICESHIPS (NAO/SOLAS AND NAO/HEA) Nature of the programme The National Apprenticeship Office (NAO) was established in 2022 from the Action Plan for Apprenticeship 2021–2025 to coordinate and drive apprenticeship expansion in Ireland. The NAO reports to a joint management board with the CEOs of SOLAS and the Higher Education Authority (HEA). Apprenticeship is a work-based learning programme where an apprentice is directly employed by an approved employer in their field of training and education. Currently, 77 apprenticeships are offered under two models: craft (25) and consortia-led (52) apprenticeships. SOLAS is the coordinating provider for all 25 craft apprenticeships. The coordinating providers for the current 52 consortia-led apprenticeship programmes consist of education and training boards, higher education institutions and designated agencies (e.g., Retail Ireland Skillnet, Accounting Technicians Ireland, FasttrackintoIT). Objectives Employment, education progression, skills enhancing across the various sectors where there are apprenticeships. Target group Unemployed, employed, individuals from socio-economically challenging backgrounds (carers, travelers, etc.). Key outcomes/KPIs Individual: Progression to employment, educational attainment; earning/occupational upgrading (primary); Firm: productivity, revenue, no. of employees (secondary). Data collected to date The NAO’s ACSS dataset contains information on gender, age, level of education, apprentice employer details, personal public service number (PPSN), apprenticeship type. The ACSS data can be linked to the CSO’s Educational Longitudinal Database (ELD), which through PPSNs matches datasets on learners that have completed courses or programmes to other datasets that describe their outcomes in subsequent years (e.g., Revenue). Data are also captured on those who register but do not continue, but as apprentice applications are made via the employer, data on socio-economic factors are not directly collected. A national survey of apprentices is due for administration. Data requirements Markers needed in Live Register (for unemployment) and learner (for employee) databases, and these data linked to Revenue. Preliminary evaluation assessment Data infrastructure suitable for counterfactual impact evaluation (CIE), with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget SOLAS apprenticeships: €150.191 million (2022); €195.936 million (2023). HEA apprenticeships: €59.599 million (2022); €78.940 million (2023). Fully funded by NTF. Relevant links Programme: https://www.apprenticeship.ie/. CSO. ‘Further education outcomes – Graduation year 2020’, https://www.cso.ie/en/releasesandpublications/ep/paoqy/apprenticeshipoutcomesqualificationyear2020/. CSO. ‘Further education outcomes – Graduation years 2010–2016’: https://www.cso.ie/en/releasesandpublications/ep/p-feo/furthereducationoutcomesgraduationyears2010-2016/apprenticeships/. Source: Grantee organisation. Note: *The HEA also received temporary funding of approximately €4 million for its apprenticeship programme in 2023. Roadmap for an evaluation of the National Training Fund |70 APPENDIX TABLE 2.2 COMMUNITY EMPLOYMENT TRAINING (DSP) Nature of the programme Community employment (CE) is an employment intervention with a training element, and it also benefits the community. All CE participants are engaged in some element of service support and delivery. The training element is NTF funded, with training mainly provided by Education and Training Boards (ETBs) (childcare and health and social care sectors). Objectives To enhance the employability and mobility of disadvantaged and unemployed persons by providing work experience and training opportunities for them within their communities. Target group Unemployed, long-term unemployed, clients on some types of allowance/benefit, people with disabilities, marginalised groups. Key outcomes/KPIs Employment, movement on to previous or new Department of Social Protection (DSP) payment; educational attainment (QQI framework); ‘distance-travelled tool’. Data collected to date PPSN, date of birth, gender, age, completed years of participation, previous social welfare claim details, additional allowance details, highest educational attainment, individual learner plan. Data collected up to four months after exit from the CE programme. Training completion points are captured on the individual learner plan. Individuals can still be participating on the CE scheme after completion of a training course. Data requirements The data from CE is within the main DSP database that records jobseeker claims and payments. PPSN is collected, so linking to Revenue data for earnings, occupation, etc. data should be feasible and is needed for a counterfactual impact evaluation (CIE) – already done by the OECD 2024 study. Preliminary evaluation assessment Data infrastructure suitable for a CIE evaluation. NTF budget €3.684 million (2022); €4.023 million (2023). The training element is NTF funded, most of the funding is from DSP. Relevant links Programme: https://www.gov.ie/en/service/412714-community-employment-programme/. OECD impact evaluation: https://www.oecd.org/en/publications/2024/03/impact-evaluation-ofireland-s-active-labour-market-policies_9548c157.html. Source: Grantee organisation. 71 | Appendices APPENDIX TABLE 2.3 EMPLOYEE AND CONTINUING PROFESSIONAL DEVELOPMENT – NEARLY ZERO ENERGY BUILDING-NZEB (SOLAS) Nature of the programme Pre-employment and continuing training for individuals post-apprenticeship and for those who are already working in the construction sector; generally, two weeks duration; provided by six ETBs. 64% of NZEB courses are non-NFQ aligned further education and training. The remaining distribution of award levels is: advanced certificate 12%, Level 5 certificate 12%, Level 3 certificate 7%, uncertified 4%. Less than 1% of courses are at Level 4. Objectives Upskilling; learn how to construct buildings to meet new NZEB requirements and learn skills to build homes that are more energy efficient, environmentally friendly and cheaper to heat. Target group Workers in construction Key outcomes/KPIs Certification/educational attainment. Data collected to date Age, gender, PPSN, educational attainment, principal economic status, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements NZEB data are collected within SOLAS’ database, Programme and Learner Support Systems (PLSS). The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget €5,175,000 (2023) Relevant link Programme website: https://www.thisisfet.ie/nzeb/. Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |72 APPENDIX TABLE 2.4 EMPLOYEE AND CONTINUING PROFESSIONAL DEVELOPMENT – SKILLS FOR WORK (SOLAS) Nature of the programme Part-time education and training initiative, provided by ETBs and sometimes in the workplace, designed in a flexible way to meet the needs of employer and employees. Courses with literacy and numeracy elements, communications, computing, interpersonal skills, problem-solving and report writing. Objectives Providing educational training opportunities to help employees in the workplace; raising the competency levels of those with low levels of educational qualifications, enhancing their communication and basic IT skills, and enabling them to cope with frequent and ongoing changes in work practices. Target group Employees in fullor part-time work, with low/outdated/no educational qualifications. Key outcomes/KPIs Course completion; educational attainment (Levels 2 and 3), but only 25% of courses are certified by QQI–FE. Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available) Data requirements Skills for Work data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget Total spend TBC Fully funded by NTF. Relevant link https://www.fetchcourses.ie/courses/parttime Source: Grantee organisation. 79 | Appendices APPENDIX TABLE 2.11 TRAINEESHIPS (SOLAS) Nature of the programme SOLAS’ traineeships are developed and delivered by the ETBs working in partnership with industry representatives and employers. They combine classroom and online training, along with experience in the workplace. They are provided across many areas, such as business, care, construction, engineering, ICT, hospitality, retail, etc. Objectives Pre-employment training; continuing training. Target group Unemployed, employed. Key outcomes/KPIs Progression to employment; Educational attainment; Progression to further learning Data collected to date PPSN, date of birth, gender, address, nationality are captured for all applicants at application stage (FETCH). Once an application is successful other data fields are collected (gender, date of birth, nationality, education level and economic status before course start, welfare status, residency status …). Traineeship data are on the PLSS database from SOLAS. Data requirements PLSS data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, within the ELD. NTF budget €2.900 million (2022 and 2023). Fully funded by NTF. Relevant link Programme: https://www.solas.ie/programmes/traineeship/. Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |80 APPENDIX TABLE 2.12 TRAINING PEOPLE FOR EMPLOYMENT – BLENDED TRAINING (SOLAS) Nature of the programme Blended training courses involves pre-employment training (although learners may require some previous experience/qualifications), with traditional face-to-face instruction and web-based online learning, provided by ETBs. Objectives Employment; further education and training. Target group Apprentices. Key outcomes/KPIs Employment; educational attainment (NFQ Levels 3 to 5). Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements Blended training data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget €1.882 million (2022); €1.065 million (2023) Relevant link NA Source: Grantee organisation. 81 | Appendices APPENDIX TABLE 2.13 TRAINING PEOPLE FOR EMPLOYMENT – BRIDGING/FOUNDATION (SOLAS) Nature of the programme Training programmes provided by ETBs. The duration is determined by the requirements of each set of skills to be acquired for the occupation (usually less than six months). Work experience at the end of the training. Objectives Training programmes are intended to bridge the gap in a person’s educational development, bringing them from a low level to a higher level. They are designed to build bridges to further training/education or employment and in the process considerably strengthen links with employers. Target group Unemployed; long term unemployed; socially disadvantaged individuals; early school leavers. Key outcomes/KPIs Educational attainment (NFQ Levels 4–6) Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements Bridging data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Moreover, data for the unemployed could be mapped to the DSP’s Live Register in order to draw an adequate control group. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. Alternatively, use of the WWLD by DSP. NTF budget € 836,000 (2022); € 1.479 million (2023) Relevant link Programme: https://www.fetchcourses.ie/courses/fulltime. Source: Grantee organisation. APPENDIX TABLE 2.14 TRAINING PEOPLE FOR EMPLOYMENT – EVENING TRAINING (SOLAS) Nature of the programme Pre-employment and continuing training provided by ETBs. Median course duration 8 weeks (minimum 1 day, maximum 55 weeks), two evening a week. Objectives To provide learners with a range of employability-related skills and qualifications to facilitate those entering the labour market for the first time. Reskilling/upskilling for those who are interested in updating or adding to their skills in their spare time. Target group Employed; unemployed (in receipt of any payment from DSP). Key outcomes/KPIs Employment; educational attainment (mainly NFQ Levels 5–6, but also Levels 1–4); progression to other further education and training. Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements Evening training data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Moreover, data for the unemployed could be mapped to the DSP’s Live Register in order to draw an adequate control group. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. Alternatively, use of the WWLD by DSP for unemployed. NTF budget € 4.025 million (2022); € 5.361 million (2023) Relevant link 2016 follow-up survey of FET programme participants: https://www.solas.ie/f/70398/x/60018bac47/followupsurveyfetprogrammeparticipants2016_final_report.pdf Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |82 APPENDIX TABLE 2.15 TRAINING PEOPLE FOR EMPLOYMENT – POST LEAVING CERTIFICATE (SOLAS) Nature of the programme The PLC programme is a full-time training programme for young people who have completed their Leaving Certificate and adults returning to education; it is provided by ETBs. Objectives Employment progression and progression to higher education. Target group 16+ who have completed the senior cycle; adults returning to education; unemployed wanting to upskill. Key outcomes/KPIs Educational attainment (Levels 5–6 on the NFQ by QQI); employment; progression to higher education. Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements PLC data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget €153.342 million (2023). NTF funding for PLC only since 2023. Relevant link Programme: https://www.plccourses.ie/. Previous evaluation: https://www.esri.ie/publications/evaluation-of-plc-programme-provision. Source: Grantee organisation. 83 | Appendices APPENDIX TABLE 2.16 TRAINING PEOPLE FOR EMPLOYMENT – RECOGNITION OF PRIOR LEARNING (SOLAS) Nature of the programme Training courses provided by ETBs, with median duration of 33 weeks (minimum 2 weeks, maximum 139 weeks). Individuals are assigned a RPL mentor, who guides them though the process of creating a RPL portfolio (i.e., evidence of past learning), that will be evaluated by a RPL assessor. Objectives RPL helps the learner get his/her prior learning formally recognised by matching their knowledge and skills to a QQI award. Target group NA Key outcomes/KPIs Educational attainment (NFQ Levels 4–6). Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements RPL data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. NTF budget €113,000 (2022); €343,000 (2023) Relevant link Programme website: https://collegeoffet.ie/rpl/#:~:text=RPL%20helps%20you%20get%20your,and%20in%20yo ur%20social%20life. Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |84 APPENDIX TABLE 2.17 TRAINING PEOPLE FOR EMPLOYMENT – SPECIFIC SKILLS TRAINING (SOLAS) Nature of the programme Pre-employment training provided by ETBs, targeting all sectors. Short courses (4–10 weeks), which usually lead to minor awards, or long courses of 6 months or more. Broad range of courses available covering hard and soft skills. Courses at different levels (NFQ Levels 3-6) QQI certified, with some uncertified or non-NFQ aligned. Objectives Upskilling and reskilling. Target group Unemployed; employed; those in some form of further education and training. Key outcomes/KPIs Employment; educational attainment (NFQ Levels 3–6). Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements Specific Skills Training data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Moreover, data for the unemployed can be mapped to the DSP’s Live Register, as done by Indecon (2020) in the CIE analysis of Specific Skills Training, in order to draw an adequate control group. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. Alternatively, use of the WWLD by DSP. NTF budget €25.712 million (2022); €38.384 million (2023) Relevant link Indecon’s counterfactual impact evaluation report: https://www.solas.ie/f/70398/x/9aa70231b7/sst-independent-evaluation-_2020_indecon.pdf. Source: Grantee organisation. 85 | Appendices APPENDIX TABLE 2.18 TRAINING PEOPLE FOR EMPLOYMENT – VOCATIONAL TRAINING OPPORTUNITIES SCHEME – VTOS (SOLAS) Nature of the programme In-person training courses provide by ETBs, from basic education and training to advanced vocational training, with a wide choice of subjects. Objectives Upskilling/reskilling/meet the education and training needs of unemployed people (originally tailored for people who left school without achieving an upper secondary qualification). Target group Unemployed, at least 21 years of age and in receipt, for at least 6 months (156 days), of payment of some social welfare allowances. Key outcomes/KPIs Employment; educational attainment (NFQ Level 3–6); higher level VTOS schemes; other FET provision. Data collected to date Age, gender, PPSN, educational attainment, PES, economic sector, outcome status, early finish reason (if applicable), outcome certification (if available). Data requirements VTOS data are collected within SOLAS’ database, PLSS. The data collected by SOLAS are very comprehensive (see Chapter 4). Moreover, SOLAS’ data are linked to the CSO’s ELD, which through PPSN matches datasets on learners that have completed courses or programmes to other datasets, which describe their outcomes in subsequent years (e.g., Revenue). Moreover, data can be mapped to the DSP’s Live Register, as done by Indecon (2020) in the CIE analysis of VTOS, in order to draw an adequate control group. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation, with use of ELD database permitted by the CSO to construct a control group for comparison. Alternatively, use of the WWLD by DSP. NTF budget € 50.333 million (2022); € 52.478 million (2023) Relevant link Indecon’s counterfactual impact evaluation report: https://www.solas.ie/f/70398/x/91bd5a18cc/independent-indecon-vtos-report-to-solas.pdf. Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |86 APPENDIX TABLE 2.19 TRAINING SUPPORT GRANTS (DSP) Nature of the programme Once-off grant to meet a short-term skills gap or training need that cannot be provided by a state provider within a reasonable time. It is not intended to substitute for training and activation measures that are funded under other programmes and agencies. Objectives Skills provision; progression to employment. Target group Unemployed, other benefit recipients, people with disabilities, marginalised groups (demand-led scheme determined on a one-to-one basis by case officers). Key outcomes/KPIs Employment; educational attainment (Level 6 on the NFQ by QQI). Data collected to date Personal information of customers is captured as part of their engagement with employment services staff. Main data are captured prior to referral to training support grants but information on the customer's journey is captured throughout the training. Data requirements The data from TSG are linked to the main DSP database that records jobseeker claims and payments. PPSN is collected, so linking to Revenue data for earnings, occupation, etc. data should be feasible and is needed for a CIE. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation. NTF budget €2.352 million (2022); €2.634 million (2023). Fully funded by NTF. Relevant links Programme: https://www.gov.ie/en/publication/0a962-operational-guidelines-training-supportgrant/#the-training-support-grant-scheme. Source: Grantee organisation. 87 | Appendices APPENDIX TABLE 2.20 WORK PLACEMENT EXPERIENCE PROGRAMME (DSP) Nature of the programme 26 weeks programme, to include 30 hours work experience per week and 60 hours of training over the 26 weeks period. Objectives - Keep jobseekers close to the labour market; - Provide those who never had a job opportunity to gain work experience and training to assist in gaining employment; - Provide those who wish to change careers an opportunity to gain work experience. Target group Unemployed, other benefit recipients, people with disabilities, marginalised groups. Key outcomes/KPIs Employment, movement on to previous or new DSP payment. Data collected to date Gender, age, educational attainment, previous employment and unemployment history, job seeking and training progress, programme participation including sector of employment, prior payment, details of host, training hours, exit reasons and outcomes. Data requirements The data from the programme are linked to the main DSP database that records jobseeker claims and payments. PPSN is collected, so linking to Revenue data for earnings, occupation, etc. data should be feasible and is needed for a CIE. Preliminary evaluation assessment Data infrastructure suitable for CIE evaluation. NTF budget €819,000 (2022); €835,000 (2023). The training element of the Work Placement Experience Programme is NTF funded (~30 per cent of the full programme), then DSP funding. Relevant links Programme: https://www.gov.ie/en/service/95fe1-work-placement-experience-programme/. Source: Grantee organisation. Roadmap for an evaluation of the National Training Fund |88 FIRM-LEVEL INITIATIVES APPENDIX TABLE 2.21 IDA IRELAND – TRAINING GRANTS TO INDUSTRY Nature of the programme IDA Ireland manages training grants to industry for its portfolio of c. 1,800 multinational client companies. Training plan proposals, covering three years, are submitted for grant aid. Objectives For employees, training aims to improve their skills, increase productivity, provide life-long learning opportunities; and increase opportunities for career progression. For firms, training aims to support the delivery of strategic initiatives; drive productivity, embrace new technologies; and secure the future viability of multinational companies in Ireland. Target group Employees and firms in the IDA Ireland portfolio of multinational client companies. Key outcomes/KPIs Outcomes/KPIs are company specific (e.g., increase revenue, employee retainment, increase automation across the business). Employees level: increase in skills and competencies; enhanced career progression. Data collected to date None on learners: No template exists for data collection among the learners. For all assisted firms, IDA Ireland captures information on finance (e.g., total revenues, gross profit, R&D spend) and employment (e.g., no. of employees, temporary employment, no. of apprentices) in Irish operation, training plan with the expected impact/outcome, plan of expected expenditures, business outcomes to achieve and how. External evaluators collect ‘post-training’ data to establish if the milestones in the training plan submitted were addressed, before grant can be drawn down. Application and validation data are captured in a same system (confidential data). Company registration numbers are collected. Data requirements Ideally, need data on participating and non-participating firms, including employees, that contain important control (e.g., firm size and sector information for firms; gender, education attainment, etc.) and follow-up information (e.g., Revenue). Preliminary evaluation assessment Significant gaps and challenges exist for CIE evaluation. However, it may be possible to use IDA Ireland data to look at impact of these grants compared to other types of grants on outcomes (employment, productivity), i.e., differential impact analysis. IDA Ireland only captures information on assisted firms. Using company registration number markers to link to Census of Industrial Production and Annual Services Inquiry to construct a control group for CIE (may be possible to measure CIE if non assisted firms are in the IDA Ireland data). Significant data protection issues would need to be solved before attempting the two strategies. NTF budget €3 million (2022 and 2023). Additional funding from DETE. Relevant link Programme: https://www.idaireland.com/training-grants. Source: Grantee organisation. Appendix I | 95 Economic & Social Research Institute Whitaker Square Sir John Rogerson’s Quay Dublin 2 Telephone: +353 1 863 2000 Email: [email protected] Web: www.esri.ie ' * An Institiúid um Thaighde Eacnamaíochta agus Sóisialta Cearnóg Whitaker Cé Sir John Rogerson Baile Átha Cliath 2 Teileafón: +353 1 863 2000 Ríomhphost: [email protected] Suíomh Gréasáin: www.esri.ie