Evaluation of the Impact of the Youth Service: NEET programme

Dixon, Sylvia; Crichton, Sarah Working Paper Evaluation of the Impact of the Youth Service: NEET programme New Zealand Treasury Working Paper, No. 16/08 Provided in Cooperation with: The Treasury, New Zealand Government Suggested Citation: Dixon, Sylvia; Crichton, Sarah (2016) : Evaluation of the Impact of the Youth Service: NEET programme, New Zealand Treasury Working Paper, No. 16/08, ISBN 978-0-947519-47-6, New Zealand Government, The Treasury, Wellington This Version is available at: https://hdl.handle.net/10419/205702 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Evaluation of the Impact of the Youth Service: NEET programme Sylvia Dixon and Sarah Crichton New Zealand Treasury Working Paper 16/08 December 2016 DISCLAIMER The views, opinions, findings, and conclusions or recommendations expressed in this Working Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury or the New Zealand Government. The New Zealand Treasury and the New Zealand Government take no responsibility for any errors or omissions in, or for the correctness of, the information contained in these working papers. The paper is presented not as policy, but with a view to inform and stimulate wider debate. NZ TREASURY WORKING PAPER 16/08 Evaluation of the Impact of the Youth Service: NEET programme MONTH / YEAR December 2016 AUTHORS Sylvia Dixon 1 The Terrace PO Box 3724 Wellington New Zealand Email Telephone Fax [email protected] (04) 890 7251 (04) 495 6686 Sarah Crichton 1 The Terrace PO Box 3724 Wellington New Zealand Email Telephone Fax [email protected] (04) 890 7234 (04) 495 6686 URL Treasury website at December 2016: http://www.treasury.govt.nz/publications/research-policy/wp/2016/16-08 ACKNOWLEDGEMENTS We would like to thank Marc de Boer for his help with the Youth Service data and Michelle Bly, Marc de Boer, Gulnara Huseynli, Dean Hyslop, and Judd Ormsby for their helpful comments on earlier drafts of this paper. NZ TREASURY New Zealand Treasury PO Box 3724 Wellington 6008 NEW ZEALAND Email Telephone Website [email protected] 64-4-472 2733 www.treasury.govt.nz DISCLAIMERS The views, opinions, findings, and conclusions or recommendations expressed in this Working Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury, Statistics New Zealand, or the New Zealand Government. The New Zealand Treasury, Statistics New Zealand, Ministry of Justice and the New Zealand Government take no responsibility for any errors or omissions in, or for the correctness of, the information contained in this Working Paper. The paper is presented not as policy but with a view to inform and stimulate wider debate. The results in this report are not official statistics – they have been created for research purposes from the Integrated Data Infrastructure (IDI) managed by Statistics New Zealand. Ongoing work within Statistics New Zealand to develop the IDI means it will not be possible to exactly reproduce the data presented here. Access to the anonymised data used in this study was provided by Statistics New Zealand in accordance with security and confidentiality provisions of the Statistics Act 1975. Only people authorised by the Statistics Act 1975 are allowed to see data about a particular person, household, business or organisation. The results in this report have been confidentialised to protect these groups from identification. Careful consideration has been given to the privacy, security and confidentiality issues associated with using administrative and survey data in the IDI. Further detail can be found in the privacy impact assessment for the Integrated Data Infrastructure available from Statistics New Zealand. 1 The results are based in part on tax data supplied by Inland Revenue to Statistics New Zealand under the Tax Administration Act 1994. These tax data must be used only for statistical purposes, and no individual information may be published or disclosed in any other form or provided to Inland Revenue for administrative or regulatory purposes. Any person who has had access to the unit-record data has certified that they have been shown, have read and have understood section 81 of the Tax Administration Act 1994, which relates to secrecy. Any discussion of data limitations or weaknesses is in the context of using the IDI for statistical purposes and is not related to the data’s ability to support Inland Revenue’s core operational requirements. 1http://www.stats.govt.nz/browse_for_stats/snapshots-of-nz/integrated-data-infrastructure/privacy-impact-assessment-for-the-idi.aspx WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme i Abstract Youth Service: Not in Employment, Education or Training (or YS: NEET) is a government programme designed to encourage and assist disadvantaged 16–17 year olds to stay in education or training and improve their qualification attainment. Community organisations are contracted to undertake needs assessments and provide mentoring and support for these youth. This paper evaluates the impact of the programme on the educational retention, qualification achievement, benefit receipt, inactivity and employment rates of participating youth in the 18–24 months after they enrol in YS: NEET. Administrative data from the Integrated Data Infrastructure (IDI) is used to measure individuals’ outcomes. The impacts of the programme are estimated by comparing the outcomes of participants with those of a matched comparison group of similar youth who did not participate. We find that YS: NEET raises the educational retention of participants in the first year, by up to 9 percentage points at peak. The proportion who complete a level 2 qualification is slightly raised, by around 2 percentage points. The programme appears to raise rather than lower participants’ subsequent benefit receipt rates, and there is no improvement in their likelihood of being employed. JEL CLASSIFICATION I38 KEYWORDS Youth transitions; youth mentoring; not in employment, education or training, impact evaluation WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme ii Executive Summary The Youth Service (YS) is a government programme for 16–18 year old youth who are considered to be at risk of poor outcomes, including long-term benefit dependency. It aims to help these young people achieve a qualification at level 2 or higher and develop life skills to reduce their risk of moving on to a working-age benefit. The Ministry of Social Development (MSD) contracts community-based social service providers to work with these young people and support them to enter and remain in education, training or workbased learning. The YS programme commenced in August 2012. This paper evaluates the YS: NEET sub-programme. The ‘Not in Employment, Education or Training’ (or NEET) strand of YS is targeted at young people aged 16–17 who are not currently receiving any income support from the government, but are considered to be at risk of moving onto a benefit at a later date. Participation in YS: NEET is voluntary. The evaluation examines two main questions: how well has the programme been targeted towards youth who are at high risk of poor outcomes, and what impact does participation have on their outcomes? Longitudinal administrative data from IDI are used to measure a range of different outcomes including educational participation, educational achievement, time spent on benefits, time spent NEET, time in employment, and time in custody. To provide a longer-term picture of New Zealand’s experience with youth transition programmes, we also evaluate the impacts of an earlier programme, the Youth Transition Service (YTS). The YTS operated from 2006 until 2012, when it was replaced by the Youth Service. Targeting of YS: NEET We begin by taking all youth who started YS: NEET in the 2012–14 period and asking whether the majority of these new participants were at high risk of experiencing poor outcomes at age 18. To evaluate this, we calculate a series of risk scores for everyone who was aged 16–17 in 2012–14, representing their predicted likelihood of experiencing poor outcomes at age 18, and then compare the risk scores of the YS: NEET participants with those of other youth in the wider age group. We find that high-risk youth were more likely to be recruited to YS: NEET than low-risk youth. However, a significant proportion of the participants were at relatively low risk of poor outcomes at age 18 (based on their observed characteristics, modelled risk scores, and subsequent outcomes). Only about half of all 2012–14 participants were drawn from the highest-risk 20% of the 16–17 year resident population. About one third of all new recruits to YS: NEET were still enrolled in school when recruited, and among these high-school-student participants, targeting was particularly weak. Only about 30% were drawn from the highest-risk 20% of the youth population. Impacts of YS: NEET Our impact evaluation focuses on the youth who started YS: NEET in either 2012 or 2013 and stayed enrolled in the programme for at least three months. The time frame for assessing the impacts of YS: NEET is the first 18–24 months after enrolment. The impacts of participation are estimated by comparing the outcomes of the participants with those of a matched comparison group, made up of youth who were very similar on a wide range of measured characteristics but did not participate in YS: NEET. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme iii Higher educational participation We find that YS: NEET had a positive impact on education and training enrolment rates, although the effects were modest in size and not sustained much beyond one year. The proportion who were enrolled in education or training was 9 percentage points higher at 6 months after starting YS: NEET, 4 percentage points higher 12 months after, and not significantly different 18 months after. The increase in educational participation during the first year after starting YS: NEET was largely due to higher enrolment in tertiary programmes. We did not find any significant change in the likelihood of staying in school. Slight increase in qualification attainment The impact on qualification achievement rates was positive but very small. YS: NEET raised both the level 1 and level 2 qualification achievement rates by about 2.0 percentage points on average. Slightly more time spent on benefits Participation in YS: NEET was associated with higher rather than lower benefit receipt in the following two years, with the impact on benefit rates increasing though time. This was true for all sub-groups of participants. Thirty-one percent of YS: NEET participants were on a benefit two years after they started YS: NEET, compared with 27% of the matched comparison group. Negative impact on employment rates YS: NEET reduced participants’ employment rates in the first year (reflecting their higher rate of retention in education) and did not have any significant impact in the second year. The limited impact of YS: NEET on qualification attainment is the most likely reason for the programme’s lack of labour market benefits. The programme’s impacts on level 1 and level 2 qualification attainment rates were small (2 percentage points on average), and there was no impact on level 3 attainment. A previous study of the employment impacts of tertiary study at levels 1-3 for low-qualified school leavers (Tumen, Crichton and Dixon, 2015) found that only students who were successful in completing a qualification were more likely to be employed afterwards. Educational achievement impacts were slightly larger for higherrisk youth While the average effects of YS: NEET on qualification attainment were very small, there was some variation between sub-groups of participants. The qualification attainment impacts were slightly larger for youth who were not continuously enrolled in formal education at the time they started YS: NEET, who made up about half of all participants. On average, they had a level 2 qualification achievement rate by the end of the following year that was 6 percentage points higher than that of their matched comparisons. Very high-risk youth – defined here as those in the highest decile of the predicted risk of poor outcomes at age 18 – also experienced larger increases in educational attainment. Participation in YS: NEET was associated with a 5 percentage point improvement in the level 2 qualification attainment rate of this subgroup, which made up around 29% of all participants in 2012–14. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme iv However, these larger impacts on qualification attainment were not translated into larger or more beneficial impacts on either employment rates or benefit receipt rates. Limitations of the YS: NEET evaluation The two-year follow-up period used in this evaluation is relatively short. However, the time pattern of the impacts on education and training rates shows that these impacts were declining rather than increasing over time, so it seems unlikely we would find larger positive impacts if the follow-up period was longer. The most important limitation of this study is that the method used can’t ensure that the study population and the comparison group (which provides the benchmark for estimating impacts) are perfectly matched on all characteristics that may affect outcomes. We acknowledge there is a risk that the impact estimates given in this paper could be either too high or too low. We see them as the best available estimates in the absence of a more rigorous evaluation method, which would need to randomly allocate eligible youth to either participation in YS: NEET or a control group. Impacts of the Youth Transition Service The Youth Transition Service (YTS) was in operation from 2006 until mid 2012, when it was replaced by the Youth Service. We estimated its impacts on the youth who participated during 2008–2011. A key objective of this supplementary impact analysis was to identify whether the benefits of participation in a youth transition programme tend to be larger in the medium term (up to five years) than in the 1–2 years immediately after participation. We find that the average impacts of YTS were insignificant or slightly negative. Participants were 1–3 percentage points less likely to be studying in the follow-up period than matched non-participants, about 2 percentage points more likely to be on a benefit, 1–2 percentage points more likely to be NEET, and 1 percentage point less likely to be employed. There was no significant impact on qualification achievement. In addition, there was little change in the estimated impacts from one year after YTS enrolment to five years after: using a longer follow-up window did not materially alter the size or pattern of results. Conclusion YS: NEET raised participants’ rates of enrolment in formal education by up to 9 percentage points at peak. This positive impact on the likelihood of studying was sustained for around one year. The flow-on to qualification achievement was quite modest, however: the level 2 attainment rate was increased by around 2 percentage points. The programme did not raise participants’ employment rates, and their benefit receipt rates were slightly raised rather than lowered. Prior international research on the impacts of youth mentoring programmes has found that they either have no impacts or small positive effects on academic achievement. The results for YS: NEET are in line with this literature. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme v Table of Contents Abstract ............................................................................................................................. i Executive Summary ......................................................................................................... ii 1 Introduction ............................................................................................................. 1 2 Literature review and description of the programme ........................................... 3 2.1 Literature review ............................................................................................................. 3 2.2 Youth Service: NEET ...................................................................................................... 4 2.3 Youth Transition Service ................................................................................................. 5 2.4 Previous evaluations of the YTS and YS: NEET ............................................................ 6 3 Methods ................................................................................................................... 7 3.1 Data sources ................................................................................................................... 7 3.2 Methods used in the impact evaluation .......................................................................... 7 3.3 Models used to estimate young people’s risk of having poorer outcomes at age 18 ...................................................................................................................14 4 The targeting of YS: NEET ................................................................................... 17 4.1 Introduction ...................................................................................................................17 4.2 Programme participation rates by level of risk ..............................................................17 4.3 Risk profile of participants by their enrolment status at the start ..................................19 4.4 Cumulative rates of participation by the end of 2013 ...................................................20 4.5 Summary of findings on targeting .................................................................................22 5 Participant characteristics and activities while enrolled ................................... 23 5.1 Introduction ...................................................................................................................23 5.2 Characteristics of YS: NEET participants at the time of their enrolment ......................23 5.3 Retention in YS: NEET, studying rates and educational achievement patterns...........25 5.4 Summary .......................................................................................................................28 6 Impact estimates ................................................................................................... 29 6.1 Introduction ...................................................................................................................29 6.2 Main impact estimates ..................................................................................................29 6.3 Impacts by enrolment status at the time of starting YS: NEET ....................................33 6.4 Impacts by the predicted risk of poorer outcomes at age 18 ........................................38 6.5 Impacts by gender, ethnic group, and highest qualification ..........................................40 6.6 Impacts by duration of enrolment in Youth Service ......................................................40 6.7 Summary of findings on the impacts of YS: NEET .......................................................41 7 Impacts of the Youth Transition Service ............................................................. 44 7.1 Introduction ...................................................................................................................44 7.2 Profile of YTS participants and their educational participation .....................................45 7.3 YTS impact estimates ...................................................................................................45 8 Conclusion ............................................................................................................ 50 References ..................................................................................................................... 52 Appendix ........................................................................................................................ 53 WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 6 Youth were to be allocated to ‘customised support’ if they did not have a plan or destination following secondary school, or had significant issues or barriers placing them at risk of long-term inactivity. The YTS provider would provide intensive support, including mentoring, coaching, counselling and interventions to help transition the youth into work, education, training or some other meaningful activity. Customised support could continue after the transition to ensure the youth remained in the activity, by providing in-work or intraining support. The main differences between YTS and YS: NEET are that: • YTS provided services to a wider age group, with the majority of participants being 15–19 years old at the time of enrolment (rather than 16–17 years in the case of YS: NEET) • YTS engaged with its clients for a shorter period of time on average. Sixty-seven percent of the 2008–2011 intakes were enrolled for less than 12 months, compared with 47% of YS: NEET participants.4 • YTS providers were not given assistance by MSD to identify and locate ‘at risk’ youth. They were expected to identify the youth who needed assistance by developing relationships with schools and community organisations (who could then make referrals), and to determine a youth’s level of need through their own assessments. • YTS providers were funded on a fee-for-service basis, while YS funding uses a mixture of fee-for-service payments and payments for specific outcomes. • The target outcomes were not as clearly defined under YTS as they are under YS. These are quite significant differences. However, the vast majority of providers that held YTS contracts in 2011 continued to provide services under YS in 2012, after YTS was replaced by the Youth Service. This continuity suggests that the nature of the services provided may have evolved rather than changed abruptly. 2.4 Previous evaluations of the YTS and YS: NEET A process and outcome evaluation of the Youth Transition Service was published by MSD in 2008 (MSD, 2008). The evaluation analysed the proportions of youth who had a positive outcome at the time of exiting from YTS, such as entry to employment, enrolment in tertiary education, or a return to school, using outcome data recorded by the providers. Due to a lack of access to suitable data, the study did not attempt to measure the impact of participation in YTS (compared with the counterfactual of not participating). MSD published an evaluation of the Youth Service, covering the first 18 months of its operation, in June 2014 (MSD, 2014). This study also did not attempt to assess the impact of the NEET strand of YS. It simply reports data on participant outcomes. For example, it reports that half of the YS: NEET participants increased their number of NCEA credits within 12 months of starting Youth Service; 15 per cent met the requirements of NCEA Level 2 within their first 12 months in the Service; and 17 per cent had already met the requirements of NCEA Level 2 prior to starting the Service. 4 After excluding people who were enrolled for less than 30 days. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 7 3 Methods 3.1 Data sources The study uses data from Statistics New Zealand’s Integrated Data Infrastructure (IDI), which combines administrative data from the tax system with data collected by other government agencies and covers all persons in New Zealand. At the time the research was carried out, the main IDI data sources used in this study provided longitudinal information on individuals’: • employment and earnings over the period from 1999 to mid 2015 • benefit payments over the period from 1993 to mid 2015 • interactions with Child, Youth and Family over the period from 1993 to mid 2015 • school enrolments over the period from 2006 to mid 2015 • tertiary education enrolments over the period from 2003 to the end of 2014 • NQF-registered qualifications completed from 2006 to the end of 2014 • custodial and community sentences served with the Department of Corrections • places of residence within New Zealand; and • movements in and out of New Zealand from 1997 to mid 2015. The information on individuals’ places of residence within New Zealand is derived from several administrative sources, including the National Health Index, Primary Health Organisation enrolments, and address data held by Inland Revenue, MSD and the Ministry of Education.5 Before finalising the paper, additional data were obtained on school and tertiary enrolments in 2015 and school and tertiary qualifications completed in 2015. These data were used to update and extend the main impact estimates which are set out in Table 6 and Figure 5. We decided not to update any other results (such as the impact estimates for sub-groups) because the work was already at an advanced stage and the new data did not show any material changes in programme impacts. 3.2 Methods used in the impact evaluation 3.2.1 Study population selection criteria The study population for the YS: NEET impact evaluation comprises people who first enrolled in the NEET strand of Youth Service in either 2012 or 2013 and met various other criteria. The selection criteria, and the impact they had on the numbers of people in the sample used for the impact evaluation, are set out in Table 1. 5 Addresses are encrypted in IDI to preserve confidentiality. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 8 Table 1 –Study population selection criteria and their impact on the sample size Selection criteria N % First enrolled in YS: NEET between 1 August 2012 and 31 December 2013, without previously enrolling in the YP or YPP strands 13,848 Identity was linked to the IDI spine 12,327 89.0 Also has IRD and Ministry of Education data linkages in IDI 12,162 87.8 Aged 15–18 years when enrolled in YS: NEET 12,129 87.6 Survived until 20th birthday 12,102 87.4 Had at least one New Zealand school enrolment record during the period when they should have been enrolled in Years 9-11 11,940 86.2 Not overseas for more than 6 months in total during the 5.5-year study period, comprising 4 years before the YS: NEET enrolment date and 18 months following it 11,136 80.4 Did not attend a school that offers international qualifications (not NCEA)* 10,860 78.4 Last school enrolment record has a valid end date** 10,827 78.2 Not on a benefit in the month of enrolment in YS: NEET 10,500 75.8 Stayed enrolled in YS: NEET for at least 90 days in the 12 months after the first enrolment 9,285 67.0 Matched with at least one comparison group individual 9,081 65.6 Notes: *This restriction was only applied if the youth was aged 15 years or older when attending the school. **This restriction was only applied if more than five years had elapsed since the enrolment start date, indicating that the school had failed to supply the end date. The numbers in this table are randomly rounded. A linkage to the IDI spine is required to provide the data needed for this study. Connections to IRD and Ministry of Education identity numbers are also required to ensure we have access to individuals’ income, benefit and education and training data. We exclude a small number of people whose age when they started YS: NEET, as recorded in IDI, was less than 15 or greater than 18 (in conflict with their recorded age in the YS enrolment dataset). We also exclude a small number of people who died before their 20th birthday, who may have incomplete outcome data for the follow-up period. We exclude people who did not enrol in a New Zealand school at least once during the period when they should have been attending Year 9 to Year 11, and those who were overseas for more than 6 months in total during the main study period (comprising the 4 years leading up to the YS: NEET enrolment date and the 18 months following it). We also exclude individuals who, when aged 15 or older, attended a school that offers qualifications that are not part of the National Qualifications Framework (such as the International Baccalaureate), because there is no information in IDI about those non-NQF qualifications. We exclude a small number of people whose last school enrolment record had a missing end date after five years (preventing us from identifying when they left school). We also exclude youth who were on a benefit in the reference month, because these youth should have been enrolled in the Youth Payment or Young Parent Payment strands of YS (even though the YS enrolment data do not show that this was the case). WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 9 Finally, we exclude youth who were enrolled in the YS for less than 90 days during the first year of enrolment, because they may not have received enough assistance to experience a measurable change in outcomes. (Note, we do not require that they were enrolled for 90 days consecutively, although the vast majority of our study population members were.) The figures in the right-hand columns of table 1 show the impact of these restrictions. Of the approximately 13,848 individuals who first enrolled in YS: NEET during 2012 or 2013, around 12,160 or 88% could be linked by Statistics NZ to the IDI spine and the Education and Inland Revenue domains in IDI. The most likely reason for linking failure at this point is that the individual’s names or birthdate were inaccurately or incompletely recorded. A further 12% were dropped from the potential study population as a result of all the other study selection criteria, leaving 10,500 persons. Of these, we then retain the 88 percent who stayed enrolled in YS for at least 90 days, giving 9,285 potential study group members. Of this latter group, 98% could be matched to at least one suitable matching non-participant and were therefore retained in the impact analysis sample (and used to produce the estimates in this paper). The final sample for the impact analysis represents 66% of the individuals who first enrolled in YS: NEET in 2012 or 2013. Although this means the sample does not include about one-third of all youth who (appear to have) enrolled in YS: NEET at some point in 2012 or 2013, the restrictions are designed to ensure that we evaluate the impacts of the programme as fairly as possible. Removing individuals with incomplete data or enrolment periods of less than 90 days should increase the likelihood of detecting programme impacts, if there are any. No data are available on the characteristics of the YS: NEET participants who were not linked to IDI. However, we compared the characteristics of the participants who were linked to IDI but excluded from the final study sample (approximately 2,877) with of those of the participants who were retained (approximately 9,285). We found that the youth who were excluded were much more likely to be 18 years at the time of their enrolment. It is possible that these 18 year olds were more likely to leave the programme (or have their enrolment terminated by their provider) in less than three months. In other respects, the youth who were excluded from the final sample appear to have been slightly less disadvantaged, in terms of the indicators of disadvantage used in this paper, than those who were retained. For example, 55% of the youth who were excluded had no NCEA qualifications, compared with 62% of the final study population. Our decision to exclude youth who were overseas for long periods, enrolled at schools offering non-NCEA qualifications, or YS: NEET participants for less than 90 days, may have caused a slight bias in the final sample towards more highly disadvantaged youth. Note that our selection criteria mean most recent migrants to New Zealand will not be included. This is appropriate because their administrative data in IDI will be too limited to provide comparable measures of their characteristics and lifetime experiences. Note also that the study population for the YS: NEET evaluation includes young people who first enrolled in YS: NEET but subsequently transferred to the Youth Payment or Young Parent Payment strands. This would normally be a result of them applying for and receiving a benefit. Seventeen percent of the study population had transferred by the middle of 2015, with most moving to YP. Individuals who first enrolled in the YP strand of the Youth Service and later transferred to YS: NEET are not included in this study population. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 10 3.2.2 Outcome measures The objectives of YS: NEET are to raise participation in formal education or training, raise qualification attainment, and reduce the likelihood that a youth will move onto a benefit when they become eligible for income support at 18 years of age. YS providers receive quarterly payments for participants who are enrolled in formal education or training, oneoff payments for their achievement of a qualification, and a one-off payment for participants who are not on benefit three months after leaving the YS. Therefore enrolment rates, qualification achievement rates and benefit receipt rates are the core outcomes against which the effectiveness of the programme should be assessed. We also consider a range of other outcomes measures that may be influenced by YS participation, including whether the youth was NEET at various points in time during the follow-up period, whether they were employed, whether they received a student allowance, whether they were serving any custodial or community sentences, and their earnings growth (if employed). We also construct some additional measures of level 2 qualification attainment, benefit take-up and NEET during the three months following the young person’s 18th birthday. The outcomes of programme participants are assessed over the 18–24 months following the first YS enrolment date. We have a complete set of data covering the first 18 months after YS: NEET enrolment for most participants. Because 2015 tertiary enrolment data were not available until near the end of the project, we did not have data on tertiary enrolments for some portion of months 13–18 for a minority of participants (those who enrolled in the second half of 2013). We constructed our outcome measures using the individuals for whom the enrolment data were available. Due to the manner in which tax data are collected in New Zealand, the employment and earnings measures in IDI are available on a calendar month basis only. There are no measures of weekly earnings, hourly earnings, or hours of work in IDI. In this study, a person is classified as ‘employed’ in a given calendar month if they received any wage and salary earnings in that month (that were reported through the tax system). For consistency, we use calendar months to construct all of our measures of post-YS enrolment activity and incomes, even though some of them (such as whether or not a benefit was received) are recorded in IDI on a daily basis. For example, the ‘employment rate’ measures the proportion of people in a particular group who received wage and salary earnings (above a minimum threshold of $10) at any time during a particular calendar month. Similarly, a person is classified as ‘in receipt of a benefit’ if they received any income from one of the main income support benefits during the calendar month, and a ‘benefit receipt rate’ is the proportion of people in a particular group who received benefit income in that month. 3.2.3 Method of estimating the program impact using matched comparisons The impact of participation in YS: NEET is estimated by selecting a comparison group of youth who were as similar as possible to the individuals in the study population but did not participate. The outcomes of the comparison group individuals in the follow-up period provide the ‘counterfactual’ against which the actual outcomes of the study population members are compared. We use a combination of exact case matching and propensity score matching to select the most appropriate comparison group members for each individual in the study population.6 6 A good overview of the propensity score matching method is given in Caliendo and Kopeinig (2005). WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 11 Tables A.1 and A.2 in the Appendix show that YS: NEET participants and the potential comparison group of all 16–17 year old non-participants who met the criteria in Table 1 above were very different in their measured characteristics. In tables A.1 and A.2 we compare the characteristics of YS: NEET participants in the month they were recruited to the programme with those of all the potential comparisons who were aged 16-17 in the month of December 2013. (This time point is an arbitrarily chosen one.) The YS: NEET participants were much more likely to live in high-deprivation neighbourhoods, to have parents or caregivers who had served custodial or community sentences, and to have been supported by a benefit for a high proportion of their childhood. They were more likely to have had Child, Youth and Family (CYF) care and protection notifications, findings or placements during their childhood. They were more likely to have attended low decile schools, to have had recorded episodes of truancy, suspensions or stand-downs from school, to have left school while aged 15 or 16. They were less likely to have achieved an NCEA qualification. There are material differences on every characteristic shown. The matching process is designed to create a comparison group that is as similar as possible to the YS: NEET participant sample. The method used to construct the comparison group had three parts. In the first stage, a pool of potential comparison group members was created by selecting all youth who met the criteria listed in Table 1, with the exception that they did not enrol in YS: NEET before or during our study period (including the follow-up period) and did not enrol in any Youth Service programme before the month of selection. Note that we did not exclude individuals who moved onto YS: YP or YS: YPP at a later stage, during the followup period, because those programmes are compulsory for youth receiving benefits, and we did not want to exclude youth who went onto a benefit from either the study group or the comparison group. The characteristics, prior activities and childhood histories of these youth can be measured in each calendar month from August 2012 through to December 2013. For each person in the potential comparison group, we generated 17 monthly records corresponding to each month in this time period, and randomly assigned a reference date within the month. The characteristics, prior activities and childhood histories of each individual were then measured as at the reference date. The purpose of creating this large pool of potential control group records was to ensure we could match each person in the study population with a group of other youth whose characteristics were as well matched as possible in the reference month – the month when the study population member first enrolled in YS. In the second stage, the study population and potential control records were divided into three sub-samples, according to whether the child was still at school, enrolled at a tertiary institution, or not in either group, at the reference date. The three sub-samples were defined as follows: • ‘at school’ – enrolled at a secondary school when they enrolled in YS and stayed in school for at least one further month (meaning they had school enrolment records for three consecutive months centred on the YS enrolment month) • ‘in tertiary’ – not at school but enrolled in a tertiary education programme when they first enrolled in YS, and stayed enrolled for at least one further month (meaning they had tertiary enrolment records for three consecutive months centred on the YS enrolment month) • ‘not enrolled’ – those who were either not enrolled in school or tertiary education at all, or not enrolled for the minimum 3 month period. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 12 The logistic regressions were then estimated separately for each sub-sample. This meant that explanatory variables that were not relevant to the circumstances of children who were still at school (for example) could be omitted from the model for the ‘at school’ subsample but included in the other two models. It also allowed the parameters to differ between the three sub-samples. To run the logistic regressions, we took a random sample of 10,000 of the potential comparison group records, so that the treatment group individuals would make up a larger proportion of the total. The explanatory variables included in the models include: • The reference month (the month of first YS: NEET enrolment in the case of the study population, and a randomly assigned reference date in the case of the potential comparisons) • An indicator for having previously participated in the Youth Transition Service (a similar programme that preceded YS and was superseded by it) • Age at the reference month • Gender • Ethnic groups (using indicators for each non-European ethnic group) • Birth cohort • New Zealand Deprivation Index score for the neighbourhood that was lived in at the reference date • Region of residence at the reference date • A measure of the proportion of time the child had been supported by a parent’s benefit during their life time • Several variables capturing the youth’s lifetime care and protection history with CYF (the total number of notifications, an indicator of whether there had been any substantiated findings, and an indicator of having had a CYF care and protection placement) • A measure of the child’s total number of CYF youth justice referrals • An indicator of whether the youth’s mother or female caregiver had no qualifications (available if the mother or caregiver has received a benefit at some time in the past) • Indicators of whether the child’s parents or caregivers had ever served a custodial or community sentence (available if the parent or caregiver has received a benefit at some time in the past) • The proportion of the individual’s childhood that was spent out of NZ, up to the reference date • Characteristics of the school that the youth currently or most recently attended, including its decile and ownership type • The level of the highest qualification held at the reference date • The numbers of NCEA credits that had been completed at levels 1, 2 and 3 • An indicator of whether the child had ever been granted special education funding • Several measures of ‘disengagement’ from school, including the total numbers of stand-downs and suspensions from all schools attended and whether there were any truancy records WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 13 • The number of schools attended since 2006 • Time elapsed since the date of leaving school (where relevant) • Whether any vocational or general skill tertiary programmes had been undertaken before the reference date • Whether any tertiary qualifications had been completed before the reference date • Measures of the number of months the individual was enrolled at school, enrolled in tertiary education, employed, or NEET during the past 48 months • Measures of the number of months the individual was in receipt of a benefit, in their own name or as a partner, during the past 18 months. A full list of the explanatory variables included in the regressions is given in Table A.2 in the Appendix. Predicted probabilities of participating in YS: NEET were then calculated for all members of the treatment group and potential comparison group (not just the sub-sample of potential comparisons that was used in the logistic regressions), using the propensity scores from each regression model. These predicted probabilities are referred to as ‘propensity scores’. The third stage of the method was to match each individual in the study population with a group of comparison individuals (within each sub-sample). Matches were only made between records with the same reference month, gender, age, highest qualification level, and region of residence. We exact match cases by reference month and region to provide some degree of control for the effects of variations in the business cycle and local labour market effects. Within those exact matching constraints, each study population individual was matched to up to 10 comparison group individuals with the closest values of their propensity score, within a radius of plus or minus 0.01 or 0.03 propensity score points (the narrower band was used when there were more available matches). Fewer than 10 matches were selected if less than 10 people met these criteria. Matching with replacement was used, meaning that each comparison group individual could be matched to more than one study population member. Each matched comparison individual was assigned a weight based on the number of matches made (eg, 0.10 if the person was one of 10 matches for a particular study sample member). These weights are applied in the subsequent analysis of impacts, to ensure that the distribution of comparison group characteristics mirrors that of the study population. We dropped individuals in the study population who could not be matched with one or more comparisons. The match rates for the three sub-samples were >99% for the ‘at school’ and ‘not enrolled’ subsamples and 94% for the ‘in tertiary’ subsample. The matching method was designed to balance the average characteristics of the study population and matched comparison groups. After matching, there were no remaining statistically significant differences in variable means between the study and comparison groups, for any of the model variables.7 Although we did not exact match on every variable, the method ensured that the matched samples were very similar in terms of their demographic and regional profiles and prior employment and income support histories. 7 This was tested by re-estimating the model using the matched observations and matching weights. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 14 Once the matched comparison groups were constructed, the impacts of YS: NEET participation were estimated as the difference between the mean outcome of the study population and the mean outcome of the matched comparison group. Standard errors and confidence intervals for each impact estimate were estimated using bootstrapping methods. We used a single set of matched comparisons for each member of the study population to estimate the impacts across all outcome variables. We did not re-run or alter the matching process for different outcomes. In our final impact estimates, we also excluded individuals who were out of New Zealand in the month of interest. For example, the estimates for outcomes at 24 months after starting YS: NEET do not include people who were out of New Zealand for more than 25 days during that particular month. 3.3 Models used to estimate young people’s risk of having poorer outcomes at age 18 One of the objectives of this evaluation is to assess whether the 16 and 17 year olds who are at highest risk of future inactivity or benefit receipt have been participating in YS: NEET. To do so, we construct three different measures of ‘poor’ outcomes at age 18, model the likelihood of a young person experiencing these poor outcomes, and then score all teenagers who were aged 16–17 during the period of YS operation (our ‘target’ population) according to their estimated likelihood of experiencing each outcome. We can then compare the predicted risk scores of the YS: NEET participants with those of all 16– 17 year olds to see whether participants were drawn from the segments of the youth population that had the highest risk scores. Outlining the approach in more detail, the measures of poor outcomes at age 18 are: • Being NEET for at least 6 calendar months of the year when aged 18 • Not having obtained a level 2 qualification or higher by the end of the calendar year in which the teenager turned 18 • Receiving a benefit in at least 6 calendar months of the year when aged 18. We used IDI data for all youth who were born in 1993–95 to construct an estimation sample. The 1993–95 birth cohorts were used because they are old enough to have complete data for age 18 in IDI, enabling us to construct the above measures for everyone, but are as close as possible in birth year to our target youth population. We constructed multiple records for each individual in the estimation sample, capturing their characteristics and lifetime history at each quarter-year of age from 16 through to 17.75. The explanatory variables used in the risk models were largely the same as those used in our models of the likelihood of YS: NEET participation, and are described above in Section 3.2.3. The records in the estimation sample were divided into three sub-samples, according to whether the youth was still at school, enrolled at a tertiary institution, or not in either group, at the reference date. Logistic regressions were then estimated separately for each dependent variable and each sub-sample. Although not all of the explanatory variables were statistically significant, all were retained in the regressions to provide the best possible predictions. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 15 The coefficients obtained from the nine regressions (one for each combination of the three sub-samples and three outcomes measures) were then used, along with data on the characteristics and lifetime histories of the target sample members, to calculate three risk scores for each target sample member, representing their predicted relative risk of experiencing each outcome at 18. Note that our risk assessment models provide risk ratings for only around 80% of youth in the total 16–17 year old resident population during the period of interest. We only model the risk of the youth who met the IDI data linkage, residence in New Zealand and other selection criteria listed in Table 1, so that we have a full set of information on the variables required for the risk prediction models. Table 2 – Model fit statistics for the risk prediction models, using the area under the ROC curve Sub-sample Risk model At school In tertiary education Not enrolled Long-term NEET when aged 18 0.784 0.918 0.884 Did not achieve a level 2 qualification by end of year when turned 18 0.899 0.962 0.966 Long-term benefit receipt when aged 18 0.869 0.938 0.905 Table 2 shows the area under the Receiver Operating Characteristic (or ROC curve) for each of the risk prediction models. The ROC curve is derived by plotting the model’s sensitivity versus (1–specificity). The area under the ROC curve gives a summary measure of model performance across the range of predicted risk values. It ranges from 0.5 for a model with no predictive power to 1 for a model with perfect predictive power. The ROC statistics range from 0.784 to 0.966, indicating moderately good through to good predictive accuracy. For all three risk measures, the predictive accuracy is higher for youth who had left school than for those who were still at school. This is not surprising, as the post-school activities of 16–17 year old school leavers are likely to be correlated with their post-school activities at 18, but we do not have this information for youth who are still at school. The predictive accuracy of the models of level 2 non-achievement is also higher than that of the models of NEET and benefit receipt at 18, reflecting the use of highly predictive variables such as the number of NCEA credits already obtained. However, we use the results of the NEET risk model in our targeting analysis because this outcome measure is most closely aligned with the targeting intent of the YS: NEET programme. To further illustrate the predictive power of the risk models, we plot the actual NEET rates of youth who were 16–17 years in August 2012 over the following three years, grouping them by their predicted risk score in August 2012 from the NEET risk model (see Figure 1 panel A). Those with risk scores in the lowest 6 deciles are classified as ‘low risk’. Those with risk scores in the 7th and 8th deciles are classified as ‘medium risk’. Those in the highest two deciles as classified as ‘high’ and ‘very high’ risk, respectively. Figure 1 shows that the actual monthly NEET rates of the youth who were in the highest deciles of predicted NEET risk were much higher than those of youth who were classified as being at lower risk. There are large and consistent differences in actual NEET rates across the four predicted risk groups. The same is true when we graph the actual monthly benefit receipt rates of youth who were 16–17 years in August 2012, grouped by their predicted risk of receiving a benefit in at least 6 calendar months of the year when they were aged 18 (Figure 1 panel B). Those in the higher-risk groups had substantially higher actual benefit receipt rates. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 22 limited the hours, the true proportion who were essentially inactive in each month and could have been available to participate in the Youth Service is probably higher than shown. What we don’t know, of course, is whether they were given the opportunity to participate in YS: NEET and declined, or whether they did not know of the existence of the programme. 4.5 Summary of findings on targeting The results of this section show that the likelihood of becoming enrolled in YS: NEET in 2012-14 was positively correlated with a youth’s relative risk of experiencing poor outcomes at age 18, with high risk youth being far more likely to participate. However, a substantial minority of participants were not drawn from the higher risk groups, and the proportion who were higher-risk decreased over time. The proportion of new recruits who were in the highest two risk deciles was initially 70–80%, but it declined gradually over 2012 and 2013 to around 50% and then plateaued during 2014. Both medium-risk and low-risk youth increased their participation shares. The reduction in the strength of targeting was associated with an increase in the enrolment of youth who were still enrolled at school, who were less likely to be in the highest deciles of risk. Our cumulative total participation rate estimates suggest that in all regions of the country, two-thirds or more of the high-risk youth who could have participated in YS by the end of 2013 had either not been contacted by that time or had refused to participate. The fact that there are a substantial number of medium to high-risk non-participants means there was a sizeable pool of non-participants in every region from whom individuals with matching characteristics could be selected, to form a matched comparison group. Unfortunately, we are not able to distinguish those who refused to participate from those who were simply not contacted. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 23 5 Participant characteristics and activities while enrolled 5.1 Introduction We summarise the characteristics and schooling history of YS: NEET participants in this section of the paper. We show that the matched comparison groups are well matched with the study population members in terms of these measured characteristics and experiences. We also provide information on the nature of the education or training that was undertaken by YS: NEET participants while they were engaged in the programme or afterwards, and the qualifications they completed. 5.2 Characteristics of YS: NEET participants at the time of their enrolment Tables A.3–A.5 in the Appendix summarise the characteristics and childhood experiences of our final matched sample of YS: NEET participants. There are four columns of data for the study population, giving results for those who were enrolled at school, enrolled in tertiary education, and not enrolled at the time of their recruitment to YS: NEET, and the total. Comparable results for the matched comparison group are given in the right-hand columns of the tables. The vast majority of YS: NEET participants were aged 16 or 17 at enrolment, but 5% were aged 15 and 1.6% were aged 18. The gender split was roughly even, although participants who enrolled in YS: NEET while still at school were more likely to be female. Counting all ethnic group affiliations that have been recorded in IDI, 42% identified as European, 45% as Maori, 18% as one of the Pacific ethnicities, and 3% as Asian. The low proportion of Asians may be partly due to our study design.9 The residence data indicates that participants tended to come from poorer socioeconomic backgrounds. About 46% were living in a neighbourhood with a New Zealand Deprivation Index score of either 9 or 10 (the most deprived deciles). We estimate that 24% had a mother or female caregiver who did not have any formal qualifications – although data quality limitations mean this measure is not considered to be very accurate.10 Nearly 20% had a parent or caregiver who had served a custodial sentence at some earlier time, and 41% had a parent or caregiver who had served a community sentence. These measures of parental corrections history are also subject to data limitations and should be treated as indicative only. Twenty-eight percent of participants had spent three-quarters or more of their childhood (from birth to the birthday before their YS: NEET enrolment) living with a parent or caregiver on a benefit. Forty-seven percent had spent at least half of their childhood with a parent/caregiver who was receiving a benefit. Fifty percent had been the subject of at least one CYF care and protection notification during their childhood. Twenty-four percent 9 We do not include recent migrants to New Zealand because the administrative data in IDI does not fully capture their characteristics and past activities. 10 The education of the mother is only known if the mother has received a benefit at some time in the past, and the educational attainment of beneficiaries has not always been accurately recorded in the benefit records. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 24 had been the subject of at least one substantiated CYF care and protection finding, and 7% had had a CYF care placement. Eleven percent of participants had had at least one referral to CYF youth justice during their childhood. Twenty-eight percent had used mental health or addiction services in the secondary health care sector at some stage during their childhood. A comparison of the data for the three enrolment sub-groups indicates that the ‘at school’ subgroup was the least disadvantaged of the three in terms of these measures of socioeconomic status and childhood hardship, while the ‘not enrolled’ subgroup was the most disadvantaged. Summary measures of school characteristics and educational achievement are set out in Table A.4, focusing on the current or most recently attended school. Most YS: NEET participants were attending or had attended low decile schools, but a minority were at, or had been at, high decile schools. Thirteen percent had attended five or more different schools since 2006, which is considerably higher than the expected 2-3 and represents a high level of movement between schools. Two percent had received special education funding at some time during their schooling. Special education funding is provided to assist children with disabilities. Other results in Table A.4 show that about 28% had a truancy record, 37% had had at least one stand-down from a school and 15% had had at least one suspension from a school. Of the participants who had left school before they enrolled in YS: NEET (the ‘in tertiary’ and ‘not enrolled’ subgroups), about a quarter had left school at 15 years or younger (before the official minimum school leaving age), and half at 16 years. Due to data limitations, we know the calendar years in which qualifications were obtained but not the exact timing within the year. Sixty-three percent held no formal qualifications in the year before their YS enrolment. The rest had NCEA level 1 (23%), NCEA level 2 (12%), or a tertiary qualification (2%). Nearly all of the tertiary qualifications were National Certificates at level 1 or level 2. The youth who began YS: NEET while they were still at school were more likely to have obtained NCEA level 2 already (17%) than those who had left school before enrolling in YS: NEET (about 10%). Others would have been working towards NCEA level 2 during the year that they enrolled in YS. The youth who enrolled in YS: NEET while they were still at school also tended to have lower school mobility and fewer past school infringements than the other two sub-groups. Table A.5 presents data on other activities that were undertaken before participation in YS. The majority of the ‘in tertiary’ and ‘not enrolled’ participants had left school within the past 12 months. Twenty-six percent of all new YS: NEET participants had been enrolled in a tertiary education programme in the past 18 months (including tertiary courses that can be undertaken at the same time as attending school). Fifteen percent of youth in the ‘in tertiary’ group and 27% of those in the ‘not enrolled’ group had been ‘NEET’ for more than 6 months of the past 18. Finally, comparison of the data on the two sides of tables A.3 to A.5 shows that the YS: NEET participants and matched non-participants are very similar on all of the indicators shown. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 25 5.3 Retention in YS: NEET, studying rates and educational achievement patterns Statistics on participants’ enrolment and achievement patterns are given in Table 4. About two-thirds of the study population remained in the YS scheme for more than one year. Another 26 percent stayed in the scheme for between 6 months and one year. These figures include the enrolment time of individuals who switched from YS: NEET to the YP or YPP options. Recall that the youth who stayed enrolled in YS for less than 90 days are not included in our study population. Table 4 – YS: NEET programme retention and education or training undertaken At school (%) Enrolled in tertiary study (%) Not enrolled (%) Total (%) N 3,225 1,635 4,224 9,084 Time enrolled in YS in the first 18 months 3-<6 months 10.5 13.0 17.0 14.0 6-<12 months 26.0 33.0 30.5 29.3 12-18 months 63.5 54.1 52.5 56.7 Education or training undertaken while participating in YS: NEET Some school attendance 100.0 10.8 19.4 46.3 Some tertiary attendance 36.2 100.0 54.7 56.3 Tertiary programmes that started before YS enrolment 5.4 98.9 8.3 23.6 Tertiary programmes that started after YS enrolment 33.8 50.3 50.8 44.6 Any school or tertiary attendance 100.0 100.0 64.8 83.6 No school or tertiary attendance 0.0 0.0 35.2 16.4 Any industry training 2.2 2.8 3.8 3.0 Any education or training 100.0 100.0 66.6 84.4 Notes: All sample size numbers are randomly rounded. Eleven percent of participants in the ‘in tertiary’ group and 19% of those in the ‘not enrolled’ group had a school enrolment during their YS participation period, indicating that they returned to school at least briefly, or perhaps enrolled at the Correspondence School. Fifty-six percent of all YS: NEET participants were enrolled in a tertiary programme at some point during their YS enrolment. This include enrolments that had been started before the student was recruited into YS. Forty-five percent started a new tertiary programme during their YS enrolment. Rates of participation in industry training were very low. About 3% of all participants and 4% of the ‘not enrolled’ subgroup were enrolled in industry training at some stage during their YS spell. The purpose of YS: NEET is to raise participation in formal education or work-based training, but 35% of youth in the ‘not enrolled’ subgroup (and 16% of all participants) did not undertake any formal education or formal work-based training while they were enrolled. These youth tended to disengage from YS sooner than other participants, but nevertheless 68% remained enrolled in the YS for more than 6 months and 29% for more than one year. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 26 Table 5 – Characteristics of the first tertiary programme after starting YS: NEET At school (%) Enrolled in tertiary study (%) Not enrolled (%) Total (%) N 1,089 825 2,148 4,056 Level Level 1 8.0 12.7 19.6 15.0 Level 2 39.4 42.2 46.5 43.9 Level 3 23.7 28.7 22.9 24.3 Level 4+ 29.2 16.4 10.8 16.8 Study load, first year Less than 0.5 EFTS 26.2 29.5 25.6 26.6 0.5-<1.0 EFTS 44.1 55.3 49.4 49.3 1.0-<2.0 EFTS 29.8 14.9 24.7 24.1 Type of provider University 15.4 1.1 2.0 5.4 Polytechnic 36.9 26.2 29.9 31.1 Wananga 5.8 4.7 7.7 6.7 Private training establishment 41.9 67.6 60.1 56.7 Funding source Student component 43.5 31.6 29.3 33.7 Youth Guarantee 44.9 64.0 65.5 59.8 Other funding 11.8 4.4 4.9 6.7 Field Natural and Physical Sciences 2.2 s s 0.8 Information Technology 4.7 6.2 6.3 5.8 Engineering and Related Technologies 9.6 5.8 7.3 7.6 Architecture and Building 6.9 6.9 6.8 6.8 Agriculture, Environmental and Related Studies 6.1 5.8 8.2 7.1 Health 4.4 1.5 2.7 2.8 Education 1.7 1.8 1.0 1.3 Management and Commerce 10.2 10.2 7.3 8.6 Society and Culture 12.1 10.2 8.4 9.8 Creative Arts 6.3 3.3 3.4 4.1 Food, Hospitality and Personal Services 9.4 16.7 12.8 12.7 Employment or life skills (mixed field) 26.7 31.6 35.6 32.5 Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. Table 5 presents data on the tertiary programmes that were started by participants after enrolling in YS: NEET. (Recall that 44% started a new tertiary programme or programmes.) If more than one programme was started on the same day, we sum the study load and select the highest qualification that was enrolled for. The majority (83%) of these tertiary programmes were aimed towards qualifications at levels 1, 2 or 3, with level 2 being most common. Three-quarters required less than one year of full-time study in the first year. Two-thirds were taught by a private training establishment (PTE) or a wananga, while the rest were mainly courses at polytechnics. Nearly two-thirds of these courses were funded through the Youth Guarantee, which provides fee-free places to eligible students. About one third (33%) were ‘mixed field’ programmes, that is, courses on employment or life skills. The remaining two-thirds were courses teaching occupationally-focused skills in fields such as food, hospitality and personal services, management and commerce, and education. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 27 Table 6 – Highest qualification by end of calendar year after year of starting YS At school (%) Enrolled in tertiary study (%) Not enrolled (%) Total (%) N 3225 1635 4224 9084 School qualification None 19.2 41.7 56.7 40.7 Level 1 10.8 11.9 13.2 12.2 Level 2 38.5 35.6 24.4 31.4 Level 3 31.6 10.6 5.7 15.8 Tertiary qualification None 81.9 33.9 67.6 66.6 Level 1 1.3 7.3 4.0 3.6 Level 2 6.2 30.5 15.1 14.6 Level 3 6.0 18.9 9.7 10.0 Level 4+ 4.7 9.4 3.8 5.1 Industry training qualification None 99.1 99.1 98.7 98.8 Level 1 s s 0.5 0.3 Level 2 0.5 s 0.6 0.5 Level 3+ 0.4 s s 0.3 All qualifications None 16.5 22.9 45.5 31.1 Level 1 10.1 10.8 12.8 11.5 Level 2 35.3 36.3 25.8 31.1 Level 3 33.4 20.6 12.1 21.2 Level 4+ 4.6 9.4 3.8 5.1 Field of highest tertiary qualification None obtained 82.0 34.9 69.0 67.4 Natural and Physical Sciences s s s s Information Technology 0.9 5.3 2.4 2.4 Engineering and Related Technologies 1.3 4.4 2.8 2.5 Architecture and Building 1.2 3.5 2.0 1.9 Agriculture, Environmental and Related Studies 1.7 4.2 3.1 2.7 Health 0.8 1.3 0.6 0.8 Education s 1.1 0.4 0.5 Management and Commerce 2.2 5.5 3.1 3.2 Society and Culture 2.0 7.9 3.1 3.6 Creative Arts 0.8 1.7 0.9 1.0 Food, Hospitality and Personal Services 2.1 11.2 4.3 4.8 Employment or life skills (mixed field) 4.7 18.5 8.7 9.0 Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. Data on the qualification attainment of the participants is given in Table 6. Because the qualification attainment data only record the year in which a qualification was completed and not the month of the year, we can’t attribute qualifications exactly to the period of YS: NEET enrolment. Instead we simply report all qualifications that were completed in the year of first YS enrolment or the following year. The studying required to obtain the qualification may have been partly or fully carried out either before or after the YS: NEET enrolment period. Starting with the highest school qualification that was obtained in the year of starting YS: NEET or the following year, 12% completed NCEA level 1 in this period, 32% NCEA level 2, and 16% NCEA level 3. Thirty percent completed a level 2 or higher tertiary qualification. Only 1 percent completed an industry training qualification at any level. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 28 Putting all types of qualification together, 57% completed a level 2 or higher qualification, 12% completed a level 1 qualification and 31% did not complete any new qualifications. Looking at the fields of the tertiary qualifications, nine percent of the YS: NEET participants completed a qualification in life skills or general employment skills and 24% completed an occupationally-oriented qualification. Nearly all of these qualifications were National Certificates. 5.4 Summary Participants in YS: NEET had many markers of potential disadvantage or risk, such as living in a low decile neighbourhood, having had past contacts with CYF, and showing signs of disengagement from school. There was some diversity in the profiles of different groups of participants, however, particularly when we compare the youth who started YS: NEET while they were still at school with those who started after they had left school. The majority had either no qualifications or just NCEA level 1 in the year before their year of enrolment in YS, but 12% had completed NCEA level 2 already. Others would have been working towards NCEA level 2 during the year they were recruited into YS: NEET. It is unclear why the youth who had obtained NCEA level 2 already were recruited into the programme. The majority (84%) of the youth in our study population undertook some form of education or training during their YS enrolment period. Forty-six percent were at school during some of the period they were enrolled in YS: NEET and 56% were enrolled in a tertiary programme. This includes tertiary enrolments that had started before the student was recruited into YS. The majority of the tertiary programmes attended were at levels 1, 2 or 3, with level 2 being most common. The majority of these programmes required less than one year of full-time study to complete. About one-third were general employment or life skills courses. Fifty-seven percent of YS: NEET participants completed a level 2 or higher qualification, and 12% a level 1 qualification, in either the year when they started YS: NEET or the following year. The other 31% did not complete any new qualifications in this period. As noted, the studying that was required to obtain these qualifications may have been partly or fully carried out before or after the YS: NEET enrolment period. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 29 6 Impact estimates 6.1 Introduction In this section we present estimates of the impact of participation in YS: NEET. The overall results for all participants in the main study population are given in Section 6.1. In section 6.3, we consider the question of whether YS: NEET impacts differ for youth who were still at school when they started YS, enrolled in tertiary study, or not studying at all. In section 6.4, we look at whether the impacts vary by the youth’s risk rating at the time they started YS: NEET. In section 6.5 we consider the variation in impacts by gender, ethnic group, and highest qualification. In section 6.6, we provide impact estimates for a broader study population that includes the youth who were enrolled in YS: NEET for less than three months, and look at variations in impact size by duration of enrolment in YS. Section 6.7 provides a summary of the findings. 6.2 Main impact estimates Our main estimates of the impacts of YS: NEET participation are summarised in Table 7 and illustrated in Figure 7. Previewing these results, they show a moderate increase in the proportion who were studying over the first year of participation, which was not sustained after 18 months, and a very small increase in the proportion achieving level 1 or level 2 qualifications. On the other hand, participants were slightly more likely to receive a benefit during the follow-up period than matched non-participants, contrary to the programme objectives, and this effect was increasing over time as an increasing proportion of those in the study population reached their 18th birthday and become eligible for a benefit. Results at the top of Table 7 show that the proportion of the YS: NEET participants who were enrolled in formal education (either school or post-school)11 was 9 percentage points higher 6 months after starting YS, and 4 percentage points higher 12 months after, than the comparable proportion of matched non-participants. About 66% of participants were enrolled in school or tertiary education after 6 months and 53% of participants were enrolled after 12 months. By eighteen months after enrolment in YS, the impact on educational participation was no longer significant. As shown in Figure 7, the higher rate of formal study was essentially due to a higher rate of tertiary enrolment. The school enrolment profiles of the YS: NEET participants and their matched comparisons were similar. Note that our measures of tertiary participation are based on tertiary enrolment records without the benefit of any data on attendance. If a student withdraws from a programme within the first few of weeks their enrolment record will be cancelled, but if they withdraw at a later stage no change is made to the administrative records and they will be counted in our estimates as ‘studying’ until the end of the enrolment period. Therefore, tertiary enrolment rates will tend to be overstated. This matters for our study results if there was a significant difference between YS: NEET participants and non-participants in the likelihood of dropping out early. If the participants were more likely to drop out before the end of their programme, our estimates of the impact of YS: NEET on studying rates are likely to be overstated. On the other hand, if 11 Because the quality of the industry training enrolment data is relatively poor, with end dates frequently not being supplied, and only 3 percent of people in the study population did any industry training, we do not construct impact measures capturing participation in any form of education or work-based training but instead focus on formal education. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 30 YS: NEET participants were less likely to drop out than the youth in the matched comparison group, our estimates of the impact of the impact on the programme on studying rates are likely to be understated. We have no evidence either way. Table 7 – Main impact estimates for all YS: NEET participants Outcome Time after starting YS: NEET N. partic. Partic. mean (%) Comp. mean (%) Impact (%) Std error (x100) Sign. Enrolled in formal education 6 months 9,063 66.2 57.6 8.6 0.67 * 12 months 9,051 53.3 49.2 4.1 0.66 * 18 months 8,952 38.6 37.9 0.7 0.65 24 months 8,877 33.1 33.0 0.1 0.57 Qualification achievement by one calendar year after starting YS: NEET NCEA Level 1+ 9,084 59.3 58.1 1.2 0.50 * NCEA Level 2+ 9,084 47.2 45.9 1.3 0.51 * NCEA Level 3+ 9,084 15.8 16.9 -1.2 0.41 Level 1+ 9,084 69.6 67.6 2.0 0.54 * Level 2+ 9,084 57.9 56.3 1.6 0.56 * Level 3+ 9,084 26.5 28.4 -1.9 0.51 * Qualification achievement by two calendar years after starting YS: NEET Level 1+ 9,078 73.9 71.6 2.3 0.55 * Level 2+ 9,078 64.0 62.0 2.0 0.61 * Level 3+ 9,078 34.5 36.9 -2.4 0.59 * Receiving a benefit 6 months 9,063 8.7 6.7 2.0 0.42 * 12 months 9,051 16.7 13.5 3.3 0.56 * 18 months 8,961 23.1 19.3 3.9 0.55 * 24 months 8,487 28.4 25.1 3.3 0.66 * Other outcomes targeted by the programme Level 2 qualification by year turned 18 7,944 59.0 57.7 1.2 0.65 Benefit receipt in the 3 months after the 18 th birthday 7,905 27.3 21.9 5.4 0.66 * Custody in the 3 months after the 18 th birthday 7,908 1.3 1.1 0.2 0.18 NEET 6 months 9,063 21.8 26.1 -4.3 0.56 * 12 months 9,051 26.8 28.6 -1.8 0.62 * 18 months 8,949 32.0 32.3 -0.3 0.71 24 months 8,877 33.8 33.4 0.5 0.60 In employment 6 months 9,063 25.0 28.3 -3.2 0.57 * 12 months 9,051 32.2 34.4 -2.3 0.63 * 18 months 8,958 38.9 39.9 -1.0 0.69 24 months 8,880 43.0 43.6 -0.6 0.63 Notes: All sample sizes are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. The sample sizes for the activity measures decline with time because we do not include individuals who were overseas for most of the month. We also exclude youth who were 18 or close to 18 when they started YS: NEET from the measures of ‘other outcomes targeted by the programme’. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 31 Figure 7 – Outcomes of all YS: NEET participants and their matched comparisons 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 036912 15 18 21 24 Percentage Months before and after starting YS Enrolled in school Comparisons Participants 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 0 3 6 9 12 15 18 21 24 Percentage Months before and after starting YS Enrolled in tertiary education Comparisons Participants 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 0 3 6 9 12 15 18 21 24 Percentage Months before and after starting YS On a benefit Comparisons Participants 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 0 3 6 9 12 15 18 21 24 Percentage Months before and after starting YS Not in employment, education or training Comparisons Participants 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 0 3 6 9 12 15 18 21 24 Percentage Months before and after starting YS Employed Comparisons Participants 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 036912 15 18 21 24 Percentage Months before and after starting YS Enrolled in Youth Service Comparisons: YS: NEET Participants: YS: NEET Comparisons: other YS Participants: other YS 0 20 40 60 80 100 -24-21-18-15-12 -9 -6 -3 0 3 6 9 12 15 18 21 24 Percentage Months before and after starting YS Enrolled in formal education Comparisons Participants 0 20 40 60 80 100 Prev year Year started YS 1st year after 2nd year after Percentage Qualifications Participants - level 1 Comparisons - level 1 Participants - level 2 Comparisons - level 2 We next show measures of qualification achievement, giving the percentages of youth who had completed an NCEA level 1, level 2 or level 3 qualification, or any type of level 1, level 2 or level 3 qualification, in the first and second calendar years after participants started the programme. NCEA achievement rates at levels 1 and 2 were slightly higher for the participants than their matched comparisons, although only by around 1 percentage point. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 38 6.4 Impacts by the predicted risk of poorer outcomes at age 18 Our measures of the predicted risk of experiencing relatively poor outcomes at age 18 can be used to explore the question of whether ‘higher risk’ youth benefit more from participation in YS than ‘lower risk’ youth. We present impact estimates for youth in four different risk groups in Table 10. The risk groups are constructed using our modelled predictions of the relative risk of being longterm NEET at 18. Recall that four risk groups were constructed after allocating all 16 and 17 year olds to decile groups according to their predicted long-term NEET risk score. The ‘low risk’ group comprises individuals in the lowest 6 deciles; the ‘medium risk’ group comprises individuals in deciles 7 and 8; the ‘high risk’ group comprises individuals in decile 9 and the ‘very high’ risk group those in decile 10. Nineteen percent of the YS: NEET participants in our study population were in the low risk group, 22% in the medium risk group, 25% in the high risk group and 34% in the very high risk group. To obtain the impact estimates reported in Table 10, we exactly matched participants and comparisons within these four risk groups as well as by the other variables included in the matching procedure. The results in Table 10 are restricted to 16 and 17 year olds – the small number of 15 and 18 year olds in the study population and comparison groups were dropped. The results show that YS: NEET participants in the ‘low risk’ group gained little benefit from participation. A small positive impact on educational participation was short-lived. Most of the estimated impacts on enrolment rates and educational achievement were small and non-significant, with the exception of benefit receipt rates, which were slightly raised. The results for participants in the ‘medium risk’ group are substantively the same as for those in the low risk group. Teens in the ‘high risk’ group had higher educational enrolment rates after starting YS, by 10 percentage points at 6 months and 5 percentage points at 12 months. But their educational achievement rates were not significantly higher, and members of this group has significantly higher benefit receipt rates in the follow-up period than their matched comparisons (by 5 percentage points at 12 months after starting YS). Only participants in the ‘very high risk’ group (representing the top decile of risk) appear to have benefitted in the manner intended by the programme. Their enrolment rates were approximately 14, 9 and 5 percentage points higher at 6, 12 and 18 months after starting YS. Level 1 and level 2 achievement rates were approximately 6 and 5 percentage points higher than those of the matched non-participants in the year after YS enrolment. However, the proportion who were on a benefit was also around 3 percentage points higher than the comparable proportion of matched non-participants. Employment rates were also lower during the two-year follow-up period. We ran similar impact estimates using the ‘benefit receipt at 18’ risk measure and these were substantively similar. WP 16/08 | Evalu ation of the impact of the Youth Service: NEET programme 39 Table 10 – Impacts by risk group, for 16–17 year olds Outcome Time after starting YS: NEET Low risk Medium risk High risk Very high risk Partic mean Impact Std E (x100) Sign. Partic mean Impact Std E (x100) Sign. Partic mean Impact Std E (x100) Sign. Partic mean Impact Std E (x100) Sign. (%) (%) (%) (%) (%) (%) (%) (%) Enrolled in formal education 6 months 72.3 3.1 1.3 * 74.3 4.1 1.5 * 69.6 9.9 1.6 * 54.5 14.3 1.7 * 12 months 62.4 0.9 1.2 61.8 0.8 1.4 52.4 4.8 1.4 * 44.0 8.5 1.7 * 18 months 43.7 -2.8 2.1 43.2 -4.7 2.0 * 37.3 0.6 1.9 34.6 5.1 1.8 * Qualification achievement by calendar year after starting YS Level 1+ 93.0 -0.3 0.6 85.6 -0.1 0.9 69.2 1.3 1.3 46.9 5.7 1.4 * Level 2+ 84.1 -1.8 0.9 75.3 -0.4 1.2 55.3 0.4 1.4 35.3 4.6 1.3 * Level 3+ 52.6 -1.6 1.3 34.4 -1.9 1.2 22.5 -2.2 1.4 11.5 -3.1 0.8 * Receiving a benefit 6 months 3.3 1.3 0.5 * 5.6 2.8 0.7 * 9.1 2.1 0.9 * 13.3 1.2 0.9 12 months 5.5 1.6 0.8 * 10.6 3.5 0.9 * 19.0 5.2 1.1 * 25.8 3.1 1.3 * 18 months 8.0 2.8 0.8 * 14.6 4.6 1.1 * 25.0 4.4 1.5 * 35.3 3.1 1.3 * Other outcomes targeted by the programme Level 2 qualification by end year turned 18 83.2 -0.9 1.3 75.9 -0.9 1.6 57.0 -1.0 1.9 36.2 4.7 1.7 * Benefit receipt in the 3 months after 18 th birthday 7.6 2.3 1.0 * 16.2 5.6 1.2 * 32.7 7.6 1.7 * 42.0 6.4 1.6 * NEET 6 months 5.8 -0.4 0.7 12.7 1.4 1.0 20.2 -5.6 1.2 * 38.0 -10.3 1.3 * 12 months 8.1 0.8 0.9 16.0 1.2 1.0 28.5 -1.8 1.4 43.0 -5.7 1.2 * 18 months 8.8 -0.1 1.2 18.9 1.4 1.7 34.0 1.4 2.0 46.4 -3.5 1.5 * In employment 6 months 55.2 0.4 1.7 27.5 -5.1 1.6 * 18.3 -3.6 1.6 * 11.3 -2.9 0.9 * 12 months 58.2 -0.5 1.5 37.0 -2.7 1.6 26.4 -4.5 1.7 * 17.2 -2.0 1.1 18 months 63.9 1.2 1.5 43.7 -4.0 1.6 * 34.3 -0.7 1.7 22.6 -0.7 1.2 Notes: ‘Low risk’ means the lowest 6 deciles in the predicted risk of being long-term NEET at age 18. ‘Medium risk’ means being in the 7th or 8th deciles, ‘high risk’ means being in the 9th decile and ‘very high risk’ means being in the top decile. The (rounded) sample sizes are as follows: 1,515 low risk, 1,776 medium risk, 2,001 high risk, 2,691 very high risk. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 40 6.5 Impacts by gender, ethnic group, and highest qualification A breakdown of the impacts by gender is given in Table A.6. The impacts estimated for males are slightly larger than those for females. For example, the increases in level 1 and level 2 achievement rates associated with YS: NEET participation are nearly 3 percentage points for males, but smaller and not statistically significant for females. Participation in YS: NEET was associated with larger increases in studying and reductions in NEET rates for males than for females, but also larger increases in benefit receipt rates. The fact that the impacts were larger for males may be partly due to a higher proportion of male participants having left school at the time they started YS. As shown above, we did not find positive impacts on average for youth who were recruited when they were still at school. A breakdown of impacts by ethnic group is set out in Table A.7. To obtain these results, we used a priority ranking scheme to assign each individual to a single ethnic group, and re-selected their comparisons using exact matching by ethnic group as well as gender, age, highest qualification, region and reference month. We show results for the European, Maori and Pacific groups only. Youth in all three ethnic groups were more likely to be enrolled in formal education in the 12 months after starting YS. The impacts on studying rates were larger for Europeans and Maori than for Pacific youth. On the other hand, the impacts on educational achievement were larger for Pacific youth. We do not find statistically significant increases in the proportion of European or Maori youth who held a level 2 qualification by the end of the year after starting YS. For Pacific youth, in contrast, we find a 4 percentage point increase in this measure of achievement. Disaggregating the impacts by highest qualification in the year before the year of enrolment in YS (Table A.8), we find somewhat larger increases in both studying rates and qualification achievement rates among unqualified youth than those who held NCEA level 1 or level 2 already. We estimate that YS: NEET was responsible for a 2 percentage point increase in the level 2 achievement rate of youth who previously had no qualifications and a 1 percentage point increase in the level 2 achievement rate of youth who already had NCEA level 1. 6.6 Impacts by duration of enrolment in Youth Service So far, we have focused on a study population restricted to youth who stayed in YS: NEET for at least 90 days in the first 12 months. We now relax that constraint and estimate average impacts for participants of all durations. We also consider whether the impacts are different for different periods of enrolment. Table A.9 gives an alternative set of main results based on an expanded study sample, including those who left after fewer than 90 days. These results are similar to those reported previously, but weaker. For example, the estimated impact on the level 1 qualification achievement rate is 1.5 percentage points rather than 2.0, and the impact on the level 2 achievement rate is 1.0 percentage points rather than 1.6. Table A.10 shows how the pattern of impacts varies by the length of time the youth stayed enrolled: 0–3 months, 3–6 months, 6–12 months, and 12 months or more. Some of the people in the last group were in the programme for more than two years. The results show negative or insignificant programme impacts for those who were enrolled for less than 3 months, slightly negative but mostly insignificant impacts for youth WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 41 who were enrolled for 3–6 months, mixed results for those who were enrolled for 6–12 months, and larger positive results for those who stayed in the programme for more than 12 months. For the latter group, rates of enrolment in education were estimated to be 14, 9 and 5 percentage points higher at 6, 12 and 18 months after starting YS: NEET. The proportion with a level 2 qualification is estimated to be 4 percentage points higher by the end of the year after the year of YS enrolment. These results suggest that the programme was more likely to benefit youth who stayed in it for at least a year. Caution is needed in drawing this conclusion, however. The duration of the YS: NEET enrolment period could be an endogenous variable that is influenced by whether or not the young person chooses to continue studying. This could be the case if providers terminate the YS enrolments of the teenagers who drop out of school or leave their tertiary programmes, for example.13 If this is the case, it would be wrong to conclude that a longer YS enrolment period leads to better outcomes. The duration of the YS: NEET enrolment period is also a very blunt measure of the amount of support that was delivered through the programme. If data on the number of contacts and the type of services provided to each participant were made available in IDI in future, we could explore the question of whether those who receive more assistance experience greater benefits in a more meaningful way. 6.7 Summary of findings on the impacts of YS: NEET YS: NEET is intended to help youth who are at high risk of becoming inactive and/or moving on to a benefit when aged 18. Our analysis of the targeting of the programme showed that the proportion of new participants who were in the highest two deciles of risk (relative to all other 16-17 year olds) was initially around 70–80%, but it declined gradually over 2012 and 2013 to only around 50%, and then plateaued at this level during 2014. Thus, the targeting of the programme weakened over time. The available data in IDI show that a high proportion of the high-risk non-participants were NEET during the study period, suggesting that in principle, they could have participated in the programme. Unfortunately, we don’t have any information on whether they were offered a place and refused to participate or were not contacted. Targeting was particularly weak among the one-third of participants who were recruited to the YS while still at school. More than 70% of youth in this sub-group were not in the two highest deciles of risk. Turning to the impacts of YS: NEET, we find evidence of a positive impact on participants’ educational participation rates over the first 12 months, a very small increase in the attainment of level 1 and level 2 qualifications, and a small increase (rather than decrease) in the proportion that moved onto a benefit during the two-year follow-up period. Specifically, by comparing the outcomes of participants with those of a matched comparison group, we estimate that: •the proportion who were enrolled in formal education was 9 percentage points higher 6 months after starting YS: NEET and 4 percentage points higher 12 months after. This was mostly due to a higher rate of enrolment in tertiary programmes at levels 1–3. 13 Youth who leave the local area, for example, will probably be dis-enrolled. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 42 •the proportion who had completed a level 2 qualification was 1.6 percentage points higher one year after enrolment in YS: NEET and 2.0 percentage points higher two years after. •benefit receipt rates were raised. The proportion who were on a benefit was 2.0 percentage points higher 6 months after starting YS, 3.3 percentage points higher 12 months after, and 3.9 percentage points higher 18 months after, than the comparable proportion of matched non-participants. By 24 months after starting YS, 28 percent were on a benefit. •the NEET and employment rates of the YS: NEET participants were reduced during the first year after enrolment in the programme (reflecting the effects of higher participation in education), but these effects did not last. In the second year after starting the programme, the NEET and employment rates of participants were not significantly different from those of the matched non-participants. •there was no significant change in the proportion who were in custody during the follow-up period. The proportion who served a community sentence was slightly higher among participants than among the matched non-participants. While the average effects of YS: NEET on qualification attainment were small, there were some material differences in these impacts between sub-groups of participants. On average, those who enrolled in YS: NEET while they were still at school or already enrolled in tertiary education did not benefit from the programme. In contrast, those who were disengaged from formal education at the time of starting YS: NEET had a level 2 qualification achievement rate that was 6 percentage points higher than that of the matched non-participants by the end of the year after starting YS. Youth with a relatively high risk of experiencing poor outcomes at age 18 – and particularly those in the highest decile of risk – also show larger positive educational impacts following participation than other youth. We estimate a 5 percentage point improvement in the level 2 qualification attainment rate of youth in the highest risk decile, but no significant impact for youth in all other risk groups. Our results also suggest that youth with no qualifications when they enrolled in YS: NEET benefitted more from their programme participation, in terms of increases in subsequent qualification attainment, than youth who held NCEA level 1 or higher qualifications at the time they started. We find evidence of higher benefit receipt rates in the follow-up period for every sub-group of participants – regardless of whether or not YS was found to have positive impacts on their rates of studying or their educational achievement. We think this could be a genuine programme effect, which is likely to be a consequence of the needs assessment work carried out by YS providers (leading to better identification of income needs and eligibility) or perhaps increased contact with other youth who are on benefits (encouraging the YS: NEET participants to apply when they turn 18). Because this is an observational study and not a randomised trial, we acknowledge that the estimates in this paper could be biased by differences between the participants and matched non-participants in unobserved characteristics, such as mental health, disabilities, substance addictions, personality traits, or the motivation to learn. There are signs that unobserved characteristics could be influencing some of our results. For example, our results for the youth who started YS: NEET while they were still at school show that participation had small negative effects on tertiary enrolment and qualification achievement rates. Our results for youth who stayed enrolled in YS: NEET for less than 6 months also imply that participation reduced rather than improved educational enrolment rates and qualification achievement. Since it is unlikely that participation in YS WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 43 had these negative effects, it is more likely that these groups of participants differed from their matched comparison in terms of some unobserved characteristics that we were not able to control for in our matching strategy, such as the motivation to continue studying. For these youth, the impact estimates we report may be biased downwards, due to ‘negative selection’ on unobserved characteristics (eg, the participants had less interest in further education than the non-participants we matched them to). For other groups of participants, positive selection on unobserved characteristics could be causing upward bias in the results. In particular, young people who were recruited from the community and were not already engaged in formal education are likely to have been more motivated to undertake further study than other similar youth – or they would not have agreed to participate. If they were more motivated, they would have been more likely to return to education and gain a qualification than other youth with similar measured characteristics, even if the YS: NEET programme did not exist. For these groups, the true impact of YS: NEET on studying rates and qualification attainment may be smaller than our estimates suggest. We are not able to say whether the overall results are more likely to be biased downward or biased upward by unobserved characteristics. Only a well-designed and wellimplemented experimental study could give unbiased estimates of the true programme impacts. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 44 7 Impacts of the Youth Transition Service 7.1 Introduction The Youth Transition Service (YTS) operated from 2004 to 2012 and was broadly similar in its objectives to YS: NEET. The goal of YTS was to ensure that all 15–19 year olds were in work, education, training, or other activities that would contribute to their long-term economic independence and well-being (MSD, 2008, p1). Community organisations around the country were contracted to contact school leavers, engage with young people who were at risk of prolonged disengagement from work, education or training, and provide these at-risk youth with customised support and guidance to facilitate their reengagement in appropriate work, education or training. The YTS programme had two main streams: ‘follow-up’, intended for youth who had a plan or destination after secondary school and no significant issues putting them at risk of long-term inactivity, and ‘customised support’, intended for youth who did not have a plan or destination following secondary school, or had significant issues or barriers placing them at risk of long-term inactivity. The ‘customised support’ strand of the programme was more similar to YS: NEET than the ‘follow-up’ strand. We investigated the impacts of YTS using IDI data on young people who enrolled in the programme between 1 January 2008 and 31 December 2011. Their educational and labour market outcomes were tracked for the following 3–5 years (depending on their start date and the period of data available in IDI). Our study sample comprises all YTS participants who were successfully matched to other data in IDI and met certain other criteria, such as being aged 15–18 when they started, attending a school that offered only NCEA qualifications, being in New Zealand for most of the study period, and staying enrolled in YTS for at least 30 days. The selection criteria were almost identical to those listed in Table 1 for the YS: NEET study sample. The final YTS study sample comprises approximately 54% of everyone who enrolled in YTS between 1 January 2008 and 31 December 2011. This fraction is relatively low because one-quarter of all YTS records could not be linked to IDI, due to the poorer quality of the name and birthdate records obtained from YTS providers. The evaluation uses the same methods as were used for the evaluation of YS: NEET. We select a comparison group of youth who were as similar as possible to the individuals in the YTS study sample, but who did not participate. The outcomes of the comparison group members in the follow-up period provide the ‘counterfactual’ against which the outcomes of the study population members are compared. Propensity score matching methods were used to select the most appropriate comparison group matches for each individual in the study population. The propensity score models were essentially the same as those outlined in Section 3.2.3 above. The main difference is that we used two years of historical data on individuals’ prior education and employment rather than four years, because the period of data available in IDI was more limited. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 45 7.2 Profile of YTS participants and their educational participation Nine percent of the youth in our study sample registered with YTS but received no further assistance. Forty-six percent received ‘follow-up’ services and 45% received ‘customised support’. Summary statistics on the characteristics of the YTS participants are set out in Table A.11 in the appendix. In terms of their socio-economic circumstances, they appear to have been similarly disadvantaged as participants in YS: NEET. For example, the proportion that attended decile 1–2 schools was similar, as was the proportion living in neighbourhoods classified to categories 9 or 10 in the NZ Deprivation Index (representing the most disadvantaged neighbourhoods). One significant difference is that a higher proportion of the YTS group were still at school when recruited (nearly two-thirds), and a higher proportion remained at school afterwards. About 50% of YTS participants were enrolled at school two months after starting YTS, compared with 35% of the YS: NEET sample. A second major difference is that 23% were aged 18, compared with less than 2% of the YS: NEET sample. A third difference is that they were more likely to hold a school qualification before they enrolled: 50 percent had no school qualifications compared with 63% of the youth who were recruited to YS: NEET. In view of the evidence presented above on variations in the impacts of YS: NEET, these differences could be expected to weaken the impact of YTS compared with that of YS: NEET. Summary statistics on the activities of YTS participants while on the programme are set out in Table A.12 in the appendix. Sixty-two percent attended school while they were on YTS and 39% took some tertiary courses. Of those who enrolled in tertiary courses, more than half enrolled in level 4 or higher programmes, compared with just 17% of the YS: NEET participants who enrolled at tertiary level. The fact that YTS participants were more likely to be taking intermediate-level rather than basic tertiary programmes would lead us to expect better outcomes for YTS participants, on average, than we observed for YS: NEET participants, but not necessarily larger programme impacts. 7.3 YTS impact estimates Impact estimates for all YTS participants, regardless of what level of assistance they received, are given in Table 11 and illustrated in Figure 10 (showing monthly activities) and Figure 11 (showing qualifications achieved). The average impacts of YTS were insignificant or slightly negative. Participants were 1–3 percentage points less likely to be studying in the follow-up period than their matched nonparticipants, about 2 percentage points more likely to be on a benefit, 1–2 percentage points more likely to be NEET, and up to 1 percentage point less likely to be employed. We find no significant impact on qualification achievement. In addition, there is no real improvement in the estimated impacts from one year after YTS enrolment to five years after: using a longer follow-up window does not lead to better results. Impact estimates for the main sub-streams of YTS are set out in Table A.13 in the appendix and illustrated in Figures 12–14 below. These results show that the young people who were allocated to the ‘follow-up’ stream had somewhat better outcomes than the non-participants we matched them to. They were around 3 percentage points more likely to complete a level 2 qualification, 1–2 percentage points less likely to be on a benefit or NEET, and 1–2 percentage points more likely to be employed, during the fiveyear follow-up period. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 46 Table 11 – Main impact estimates for the YTS programme Outcome Time after starting YTS N. partic Partic. mean (%) Comp. mean (%) Impact (%) Std error (x100) Sign. Enrolled in formal education 6 months 19,812 59.0 61.0 -2.0 0.38 * 12 months 19,806 48.9 51.8 -2.9 0.37 * 18 months 19,806 45.5 47.7 -2.2 0.41 * 24 months 19,791 39.1 40.8 -1.7 0.34 * 36 months 19,569 30.8 32.2 -1.4 0.38 * 48 months 14,013 24.3 25.6 -1.3 0.45 * 60 months 8,805 18.6 20.1 -1.5 0.52 * Level 2 qualification achievement First year 19,881 61.1 60.8 0.4 0.27 Second year 19,881 67.7 67.5 0.2 0.27 Third year 19,881 71.1 71.0 0.1 0.28 Fourth year 14,724 72.7 72.4 0.3 0.35 Fifth year 9,525 73.6 72.7 0.8 0.45 Receiving a benefit 6 months 19,812 12.4 10.4 2.0 0.25 * 12 months 19,806 16.4 14.5 1.9 0.30 * 18 months 19,806 20.0 17.9 2.2 0.33 * 24 months 19,791 23.0 20.9 2.1 0.33 * 36 months 19,569 27.0 24.5 2.5 0.34 * 48 months 14,013 27.5 25.4 2.1 0.44 * 60 months 8,805 28.0 25.8 2.2 0.59 * NEET 6 months 19,812 25.9 23.3 2.6 0.37 * 12 months 19,806 28.3 25.8 2.5 0.39 * 18 months 19,806 29.8 27.2 2.6 0.37 * 24 months 19,791 29.8 28.1 1.7 0.37 * 36 months 19,569 31.0 30.0 1.1 0.37 * 48 months 14,013 31.8 31.2 0.7 0.49 60 months 8,805 33.1 31.4 1.7 0.56 * In employment 6 months 19,812 31.3 32.2 -0.9 0.38 * 12 months 19,806 36.3 37.2 -0.9 0.37 * 18 months 19,806 39.8 40.8 -1.0 0.36 * 24 months 19,791 44.2 45.1 -0.9 0.35 * 36 months 19,569 50.0 50.5 -0.5 0.39 48 months 14,013 53.9 54.1 -0.2 0.50 60 months 8,805 56.2 56.7 -0.5 0.63 Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 47 Figure 10 – Outcomes of all YTS participants and their matched comparisons 0 20 40 60 80 100 -24-19-14 -9 -4 1 6 11 16 21 26 31 36 41 46 51 56 Percentage Months before and after starting YTS Not in employment, education or training Comparisons Participants 0 20 40 60 80 100 -24-19-14 -9 -4 1 6 11 16 21 26 31 36 41 46 51 56 Percentage Months before and after starting YTS Enrolled in formal education Comparisons Participants 0 20 40 60 80 100 -24-19-14 -9 -4 1 6 11 16 21 26 31 36 41 46 51 56 Percentage Months before and after starting YTS Employed Comparisons Participants 0 20 40 60 80 100 -24-19-14 -9 -4 1 6 11 16 21 26 31 36 41 46 51 56 Percentage Months before and after starting YTS On a benefit Comparisons Participants Figure 11 – Qualification attainment of all YTS participants and their matched comparisons 0 10 20 30 40 50 60 70 80 90 100 Prev year Year started YTS 1st year after 2nd year 3rd year 4th year 5th year Percentage Participants - Level 1 Comparisons - Level 1 Participants - Level 2 Comparisons - Level 2 WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 54 Table A.1 continued – Demographic characteristics and childhood and schooling history of the study population and potential comparison group, before matching YS: NEET participants in the study sample, before matching All potential comparisons who were aged 16-17 at Dec 2013 %% CYF care and protection notifications in childhood None 49.4 82.1 1-2 22.1 10.6 3+ 28.5 7.3 CYF care and protection finding in childhood 24.0 6.7 CYF care and protection placement in childhood 6.6 1.3 CYF youth justice referrals in childhood None 88.3 98.3 1-2 7.0 1.1 3+ 4.7 0.6 Used any mental health, drug or alcohol services 28.6 10.5 Decile of last school attended 1-2 27.6 9.6 3-4 22.7 14.7 5-6 22.7 25.3 7-8 15.0 25.5 9-10 6.1 20.4 Not available 5.9 4.4 Number schools attended since 2006 1-2 36.2 43.1 3-4 50.2 52.4 5+ 13.6 4.5 Had special education funding in school (%) 2.2 1.0 Had truancy record (%) 27.9 9.4 Had standdowns from school (%) 37.4 10.7 Had suspensions from school (%) 14.9 3.2 Age when left school (prior to YS enrolment) Still at school 37.0 74.5 15 or less 16.7 2.9 16 34.3 9.7 17 12.1 12.8 Highest qualification held, end year before enrolment in YS None 62.1 43.6 NCEA Level 1 22.5 40.6 NCEA Level 2 11.9 14.4 NCEA Level 3 1.0 0.2 Tertiary qualification, eg a National Certificate 2.4 1.2 Time elapsed since last school enrolment <2mths 11.4 13.1 2-3mths 14.9 1.3 4-6mths 11.2 2.5 7-12mths 14.1 4.8 1-2yrs 10.3 3.3 2+ years 1.2 0.5 Still at school 37.0 74.5 WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 55 Table A.2 – Variables used in the propensity score regressions Variable Description Reference month Calendar month when first enrolled Previously participated in the Youth Transitions Service 0,1 Personal and family characteristics and childhood experiences Birth cohort 1993 - 1998 Age when started YS 15-18 Gender = female 0,1 Ethnic groups (multiple permitted) 0,1 indicators for Māori, Pacific, Asian, other non-European NZ Deprivation Index of the meshblock of residence 10 values, grouped into 5 quintiles Region of residence Regional council area Proportion of childhood spent overseas <10%, 10-<50%, 50-<75%, 75%+ Mother / caregiver unqualified 0,1 Parent / caregiver served a custodial sentence 0,1 Parent / caregiver served a community sentence 0,1 Proportion of childhood supported by a parent's benefit None, 1-9%, 10-24%, 25-49%, 50-74$, 75+% Number of CYF care and protection notifications in childhood 0, 1-2, 3-9, 10+ CYF care and protection finding in childhood 0,1 CYF care and protection placement in childhood 0,1 Number of CYF youth justice referrals in childhood 0, 1-2, 3-9, 10+ Used any mental health, drug or alcohol services in the secondary health sector 0,1 Schooling history before enrolling in YS Decile of last school attended Grouped into 5 quintiles Number of schools attended since 2006 1-2, 3, 4, 5, 6+ Ever had special education funding during childhood 0,1 Truancy record 0,1 Total number of stand-downs from school 0, 1-2, 3-9, 10+ Total number of suspensions from school 0, 1-2, 3-9, 10+ Type of last school attended Regular, correspondence or other Authority of last school attended State, state integrated, private, other WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 56 Table A.2 continued – Variables used in the propensity score regressions Age when left school Not applicable, 15 or less, 16, 17 Highest qualification held at end of year before YS enrolment None, NCEA Level 1, NCEA Level 2, NCEA Level 3, tertiary qualification (eg,a National Certificate) Number of NCEA credits held at level 1 0-<5, 5-49, 50-59, 60-79, 80+ Number of NCEA credits held at level 2 0-<5, 5-49, 50-59, 60-79, 80+ Number of NCEA credits held at level 3 0-<5, 5-49, 50+ Time elapsed between leaving school and starting YS: NEET Not applicable, <2 months, 2-3 months, 4-6 months, 7-12 months, 12-17 months, 18-23 months, 24+ months Other activities prior to starting YS: NEET Obtained tertiary qualification by the year before the year of enrolment in YS: NEET 0,1 Types of tertiary programme(s) enrolled in previously General skills only, occupational skills only, or both Months at school in previous 4 years Up to 4 variables capturing months of school enrolment in the previous month, 2-6 months prior, 7-18 months prior and 19-48 months prior Months of tertiary enrolment in previous 4 years Up to 4 variables capturing months of tertiary enrolment in the previous month, 2-6 months prior, 7-18 months prior, 19- 48 months prior Months of NEET status in previous 4 years Up to 4 variables capturing months of NEET in the previous month, 2-6 months prior, 7-18 months prior, 19-48 months prior Months of benefit receipt in previous 4 years Up to 4 variables capturing months of benefit receipt in the previous month, 2-6 months prior, 7-18 months prior, 19-48 months prior Months of employment in previous 4 years Up to 4 variables capturing months of employment in the previous month, 2-6 months prior, 7-18 months prior, 19-48 months prior WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 57 Table A.3 – Personal characteristics and childhood history of the matched samples At school In tertiary Not enrolled Total At school In tertiary Not enrolled Total %%%% %%%% N3,225 1,635 4,221 9,081 26,244 8,067 29,286 63,597 Year started in YS 2012 11.9 13.6 31.8 21.4 11.9 13.6 31.8 21.4 2013 88.2 86.6 68.2 78.6 88.2 86.6 68.2 78.6 Age at start of YS participation 15 7.2 1.3 3.9 4.6 7.1 1.5 4.0 4.6 16 54.1 50.5 47.8 50.5 54.1 50.5 47.7 50.5 17 38.0 45.0 46.8 43.4 38.1 45.1 46.9 43.5 18 0.8 3.1 1.6 1.6 0.7 3.1 1.4 1.5 Gender Male 42.2 50.8 52.5 48.6 42.2 50.8 52.6 48.6 Female 57.8 49.4 47.5 51.5 57.8 49.2 47.5 51.4 Ethnic groups (including multiple ethnicities per person) European 37.4 47.0 44.0 42.2 39.9 47.9 45.8 44.1 Māori 34.2 47.0 52.8 45.2 33.9 45.5 51.2 44.0 Pacific 31.8 13.0 9.7 18.2 29.5 14.7 9.6 17.6 Asian 5.7 1.5 1.7 3.1 6.8 1.8 1.6 3.5 Other 1.6 1.1 1.1 1.3 2.0 1.3 1.3 1.5 Deprivation index of neighbourhood of residence 1-2 7.7 8.6 6.0 7.1 7.2 7.5 6.0 6.7 3-4 10.6 9.7 9.2 9.8 10.0 10.6 9.2 9.7 5-6 14.0 15.4 15.1 14.8 14.8 14.5 13.9 14.3 7-8 21.1 20.0 22.8 21.7 22.3 21.7 24.1 23.0 9-10 46.2 45.7 46.3 46.2 45.5 45.7 46.4 46.0 Region of residence Northland 5.2 8.8 5.0 5.7 4.7 9.0 4.8 5.5 Auckland 53.5 29.5 20.5 33.9 55.3 31.9 20.8 35.1 Waikato 10.0 13.8 15.6 13.2 9.9 13.8 16.1 13.5 Bay of Plenty 4.1 6.2 11.1 7.7 4.0 5.7 11.2 7.6 Gisborne 0.8 2.9 5.1 3.2 0.7 1.7 5.2 3.0 Hawkes Bay 2.2 5.1 6.0 4.5 2.1 4.6 6.0 4.4 Taranaki 1.5 1.7 2.6 2.0 1.3 1.1 2.6 1.9 Manawatu-Wanganui 3.3 4.0 7.2 5.3 3.4 4.4 7.4 5.5 Wellington 3.6 6.4 7.3 5.8 3.3 5.7 7.0 5.5 West Coast 0.7 s1.0 0.7 0.6 s0.9 0.6 Canterbury 9.5 15.6 9.4 10.5 10.4 18.3 10.0 11.7 Otago 0.7 1.7 2.2 1.6 0.7 1.8 2.2 1.6 Southland 0.5 0.9 1.9 1.2 0.5 0.9 1.9 1.2 Tasman 1.8 0.7 1.8 1.6 1.3 0.7 1.6 1.4 Nelson 1.6 s1.6 1.3 1.3 s1.2 1.0 Marlborough 0.5 0.9 1.2 0.9 s s 0.8 0.4 Proportion of childhood spent overseas <10% 86.3 92.5 94.4 91.2 85.4 90.8 94.1 90.4 10-<50% 8.4 4.6 3.3 5.3 8.6 5.1 3.6 5.6 50-<75% 4.7 2.8 2.0 3.1 5.3 3.5 1.9 3.4 75%+ 0.8 s0.4 0.5 0.7 s0.4 0.4 Participants Matched comparisons Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 58 Table A.3 continued – Personal characteristics and childhood history of the matched samples At school In tertiary Not enrolled Total At school In tertiary Not enrolled Total %%%% %%%% N3,225 1,635 4,221 9,081 26,244 8,067 29,286 63,597 Mother / caregiver unqualified 20.3 25.1 26.7 24.1 20.3 25.1 26.7 25.9 Parent / caregiver served a custodial sentence 11.8 20.4 24.9 19.4 12.9 19.1 25.1 19.7 Parent / caregiver served a community sentence 29.7 43.3 49.7 41.4 30.0 41.7 49.5 41.2 Proportion of childhood supported by a parent's benefit None 25.4 15.0 11.4 17.0 24.3 14.5 10.7 16.2 1-9% 12.3 9.2 7.9 9.7 12.6 10.1 8.7 10.3 10-24% 10.6 9.7 9.2 9.8 11.3 10.3 9.2 10.1 25-49% 15.8 17.1 17.6 16.9 16.0 17.6 17.1 16.8 50-74% 16.0 20.4 21.0 19.1 17.1 20.4 22.8 20.4 75+% 19.8 28.6 33.0 27.6 18.6 27.0 31.4 26.1 CYF care and protection notifications in childhood None 63.4 46.6 40.1 49.6 63.5 49.2 41.2 50.6 1-2 18.8 23.5 24.0 22.1 18.8 23.5 24.0 22.0 3+ 17.8 29.9 35.9 28.4 17.7 27.3 34.8 27.4 CYF care and protection finding in childhood 16.9 25.3 29.1 24.1 16.8 23.5 26.8 22.7 CYF care and protection placement in childhood 4.0 6.6 8.6 6.6 3.7 6.6 7.7 6.1 CYF youth justice referrals in childhood None 96.1 87.2 83.3 88.5 97.2 88.1 85.3 90.0 1-2 2.7 7.9 9.5 6.8 1.8 8.4 8.5 6.1 3+ 1.2 5.0 7.2 4.7 0.9 3.3 6.3 3.8 18.1 34.5 33.7 28.3 18.2 34.3 31.3 27.2 Used any secondary mental health, drug or alcohol services in secondary health sector Participants Matched comparisons Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 59 Table A.4 – Schooling history and achievement of the matched samples At school In tertiary Not enrolled Total At school In tertiary Not enrolled Total %%%% %%%% N3,225 1,635 4,221 9,081 26,244 8,067 29,286 63,597 Decile of last school attended 1-2 33.7 22.8 24.4 27.4 30.4 26.4 24.1 26.8 3-4 20.7 20.7 24.7 22.6 24.8 19.6 26.2 24.5 5-6 19.2 26.4 24.6 23.0 19.5 24.4 25.7 23.3 7-8 13.8 17.6 15.1 15.1 13.7 17.6 13.3 14.2 9-10 6.0 7.9 5.5 6.1 6.2 7.3 5.3 6.0 NA 6.6 5.1 5.6 5.9 5.2 4.6 5.5 5.3 Number schools attended since 2006 1-2 39.3 34.1 35.0 36.4 39.6 35.6 33.0 35.8 3-4 51.2 52.3 48.8 50.2 50.5 48.8 51.4 50.6 5+ 9.5 13.6 16.3 13.4 9.8 15.6 15.6 13.5 Had special education funding in school (%) 1.3 2.6 2.6 2.1 1.3 3.1 2.7 2.3 Had truancy record (%) 15.0 30.8 36.5 27.8 14.6 32.7 36.2 27.9 Number of standdowns from school None 77.5 58.2 53.5 62.9 78.2 59.3 53.8 63.5 1-2 1.7 2.2 2.4 2.1 2.1 2.0 2.6 2.3 3-9 20.8 39.4 44.1 35.0 19.6 38.9 43.6 34.2 Number of suspensions from school None 93.5 81.7 80.3 85.2 94.4 83.1 80.2 85.8 Some 6.5 18.3 19.8 14.8 5.6 16.9 19.8 14.2 Age when left school (prior to YS enrolment) Not applicable 100.0 1.1 3.9 37.5 10.0 1.8 3.8 5.6 15 or less 0.0 22.9 26.4 16.4 0.0 22.9 25.8 16.1 16 0.0 59.3 50.1 34.0 0.0 60.4 51.6 34.9 17 0.0 16.7 19.5 12.1 0.0 14.5 18.4 11.2 Highest qualification held, year before enrolment in YS None 47.0 71.2 71.8 62.9 47.0 71.4 71.8 62.9 NCEA Level 1 35.9 16.5 14.6 22.5 35.9 16.5 14.6 22.5 NCEA Level 2 16.7 8.3 9.5 11.8 16.7 8.4 9.5 11.9 NCEA Level 3 s1.1 1.6 0.9 s1.1 1.6 0.9 Tertiary qualification, eg a National Certificate 0.6 2.9 2.6 1.9 0.5 2.8 2.6 1.9 Number of NCEA credits at level 1, year before enrolment in YS None / Missing 24.8 27.5 34.9 30.0 23.1 26.6 34.5 29.0 Less than 40 9.7 30.6 26.9 21.4 10.1 32.5 27.6 22.3 40-59 7.9 14.5 10.9 10.5 9.6 12.7 10.7 10.7 60-79 13.0 10.3 10.9 11.6 13.9 9.9 10.3 11.5 80+ 44.4 17.1 16.4 26.5 43.4 18.5 16.8 26.6 Number of NCEA credits at level 2, year before enrolment in YS None / Missing 62.8 66.6 68.3 66.0 62.9 65.1 68.0 65.7 Less than 40 18.5 24.4 20.1 20.3 19.3 25.0 20.3 20.8 40-59 5.5 3.9 4.8 4.9 5.0 5.1 5.4 5.2 60-79 5.0 2.4 3.6 3.9 5.3 2.2 3.6 3.9 80+ 8.2 2.6 3.1 4.8 7.5 2.8 2.8 4.5 Participants Matched comparisons Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 60 Table A.5 – Other activities of the matched samples before enrolling in YS: NEET At school In tertiary Not enrolled Total At school In tertiary Not enrolled Total %%%% %%%% Time elapsed since last school enrolment <2 months 0.0 11.9 19.0 11.0 0.0 9.0 16.6 9.3 2-3 months 0.0 21.1 23.7 14.8 0.0 19.8 22.7 14.1 4-6 months 0.0 22.2 15.4 11.2 0.0 22.2 16.7 11.8 7-12 months 0.0 26.8 19.8 14.0 0.0 30.8 21.0 15.3 1-2 years 0.0 14.9 16.3 10.2 0.0 14.3 17.3 10.6 2+ years 0.0 2.0 1.8 1.2 0.0 2.2 1.8 1.3 Not applicable 100.0 1.1 3.9 37.5 99.9 1.7 3.8 37.5 Months at school in previous 18 months None s6.2 7.0 4.4 s6.6 8.0 4.9 1-6 months 0.4 12.3 13.4 8.6 0.5 11.0 13.5 8.4 7-12 months 1.8 28.4 22.6 16.3 1.4 32.8 23.5 17.3 13-18 months 97.9 52.8 56.9 70.7 98.2 49.4 55.2 69.4 Months of tertiary enrolment in previous 18 months None 93.9 0.0 87.5 74.0 93.9 0.0 86.4 73.5 1-6 months 4.1 76.2 5.6 17.8 3.9 73.5 6.6 17.7 7-18 months 2.0 23.8 6.9 8.2 2.2 26.5 7.0 8.8 Months of employment in previous 18 months None 73.2 67.0 63.8 67.7 74.7 67.5 62.3 67.7 1-6 months 17.2 21.3 21.3 19.8 16.1 19.8 22.7 19.9 7-18 months 9.6 11.9 15.0 12.5 9.3 12.7 15.0 12.6 Months of NEET in previous 18 months None 88.3 49.4 28.0 53.3 91.1 44.2 27.6 53.2 1-6 months 10.4 35.8 45.5 31.3 7.7 40.6 45.1 31.0 7-18 months 1.3 14.9 26.6 15.5 1.2 15.2 27.2 15.8 Months of benefit receipt in previous 18 months None 99.9 99.4 99.1 99.4 99.8 99.6 98.9 99.3 1-6 months s s 0.8 0.4 s s 0.6 0.3 7-18 months sss0.0 s s 0.5 0.2 Average monthly earnings in the 6 months before YS participation None 78.9 74.5 70.4 74.2 79.9 74.7 69.3 74.0 Less than $500 10.9 10.8 9.2 10.1 10.4 9.2 7.7 9.0 $500-$1000 6.9 8.3 8.2 7.7 6.5 8.4 7.0 7.1 $1000-$2500 3.1 5.5 10.7 7.0 3.2 6.8 13.1 8.4 More than $2500 s0.7 1.6 0.9 s1.1 2.7 1.5 Nature of any tertiary enrolments before YS enrolment Occupational skills programmes only 4.2 43.5 11.0 14.4 4.1 44.4 9.1 13.7 Life skills or employment skills only 2.0 32.5 5.1 8.9 2.0 32.7 5.3 9.1 Both occupational and general 0.7 24.0 3.5 6.2 0.5 22.9 2.2 5.3 Enrolled in Youth Training before YS s3.1 4.7 2.8 s2.4 3.7 2.2 Participants Matched comparisons Notes: All sample size numbers are randomly rounded. s = suppressed for confidentiality reasons. WP 16/08 | Evaluation of the impact of the Youth Service: NEET programme 61 Table A.6 – Impacts by gender Outcome and time after starting YS: NEET N participants Participant mean (%) Impact (%) Std error (x100) Sig. N participants Participant mean (%) Impact (%) Std error (x100) Sig. Enrolled in formal education 6 months 4,398 61.9 9.1 0.9 *4,665 70.2 8.1 1.0 * 12 months 4,392 47.7 5.0 1.0 *4,659 58.6 3.3 0.9 * 18 months 2,631 34.3 2.0 1.4 2,646 44.5 0.3 1.3 Qualifications achieved by the first calendar year after starting YS Level 1 + 4,410 65.6 2.7 0.8 *4,674 73.4 1.3 0.7 Level 2 + 4,410 54.1 2.8 0.8 *4,674 61.5 0.5 0.8 Level 3 + 4,410 20.5 -1.3 0.6 *4,671 32.3 -2.4 0.8 * Receiving a benefit 6 months 4,398 7.8 2.7 0.5 *4,662 9.6 1.3 0.6 * 12 months 4,392 14.8 4.5 0.7 *4,662 18.6 2.2 0.7 * 18 months 4,362 20.0 5.2 0.8 *4,596 26.1 2.6 0.7 * Other outcomes targeted by the programme Level 2 qualification by end of year when turned 18 2,721 54.7 1.9 1.0 2,835 62.9 1.6 1.1 Benefit receipt in the 3 months after the 18th birthday 3,354 25.2 8.0 0.9*3,612 29.7 3.5 1.0 * Not in employment, education or training 6 months 4,398 23.8 -3.3 0.8 *4,662 20.0 -5.2 0.9 * 12 months 4,392 28.4 -1.6 0.9 4,662 25.2 -2.1 0.9 * 18 months 2,634 33.2 0.2 1.3 2,646 29.8 -2.1 1.2 In employment 6 months 4,398 25.8 -4.8 0.8 *4,665 24.4 -1.8 0.8 * 12 months 4,392 34.0 -3.2 0.9 *4,659 30.4 -1.3 0.9 18 months 4,362 41.4 -1.6 1.0 4,596 36.5 -0.4 0.9 Male Female Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. WP 16/08 | Evluation of the impact of the Youth Service: NEET programme 62 Table A.7 – Impacts by ethnic group European Māori Outcome and time after starting YS: NEET N participants Participant mean (%) Impact (%) Std error (x100) Sig. N participants Participant mean (%) Impact (%) Std error (x100) Sig. N participants Participant mean (%) Impact (%) Std error (x100) Sig. Enrolled in formal education 6 months 3,129 63.9 8.1 1.1 *4,008 62.4 11.1 1.0 *1,353 79.5 4.3 1.3 * 12 months 3,129 48.9 3.6 1.3 *4,008 49.2 4.8 1.1 *1,353 72.6 4.7 1.5 * 18 months 1,839 35.6 0.6 1.6 2,574 38.1 2.4 1.2 *618 54.4 0.1 2.8 Qualifications achieved by the calendar year after starting YS Level 1 + 3,129 71.1 1.7 0.8 *4,008 61.1 2.3 0.9 *1,353 85.7 2.9 1.1 * Level 2 + 3,129 57.7 0.6 1.0 4,008 48.9 1.6 1.0 1,353 79.2 4.1 1.2 * Level 3 + 3,129 26.3 -1.9 0.9 *4,008 19.6 -2.3 0.8 *1,353 42.5 0.8 1.5 Receiving a benefit 6 months 3,129 8.5 2.8 0.6 *4,008 10.4 1.5 0.7 *1,353 4.0 1.0 0.7 12 months 3,129 16.1 4.6 0.9 *4,008 20.6 3.4 0.9 *1,353 6.9 1.2 0.9 18 months 3,129 21.7 5.9 1.0 *4,008 27.9 2.9 1.1 *1,353 10.3 0.4 1.2 Other outcomes targeted by the programme Level 2 qualification by end of year when turned 18 1,959 58.4 0.3 1.3 2,433 50.0 2.1 1.3 798 81.3 4.3 1.6 * Benefit receipt in the 3 months after the 18th birthday 2,418 26.4 7.3 1.3 *3,054 34.9 5.3 1.2 *1,041 9.9 -0.7 1.3 Not in employment, education or training 6 months 3,129 19.4 -2.9 1.0 *4,008 27.1 -6.4 0.9 *1,353 13.7 -3.3 1.3 * 12 months 3,129 23.7 -1.3 1.2 4,008 33.3 -2.9 0.9 *1,353 15.5 -3.9 1.2 * 18 months 1,839 27.1 0.8 1.6 2,574 36.7 -2.3 1.4 618 21.6 -2.5 2.6 In employment 6 months 3,129 36.2 -3.0 1.4 *4,008 19.1 -3.4 1.1 *1,353 16.1 -0.9 1.3 12 months 3,129 43.8 -3.0 1.3 *4,008 25.4 -2.2 1.1 1,353 23.9 1.4 1.6 18 months 3,129 49.3 -2.9 1.3 *4,008 31.7 0.1 1.0 1,353 32.4 0.3 1.9 Pacific Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk. WP 16/08 | Evluation of the impact of the Youth Service: NEET programme 63 Table A.8 – Impacts by highest qualification in the year before enrolling in YS: NEET No qualifications NCEA level 1 NCEA level 2 Outcome and time after starting YS: NEET N participants Participant mean (%) Impact (%) Std error (x100) Sig. N participants Participant mean (%) Impact (%) Std error (x100) Sig. N participants Participant mean (%) Impact (%) Std error (x100) Sig. Enrolled in formal education 6 months 5,700 63.8 11.1 0.8 *2,037 76.8 5.1 1.1 *1,071 61.9 2.6 1.6 12 months 5,688 51.2 5.6 0.8 *2,037 64.4 1.3 1.3 1,068 47.8 0.9 1.7 18 months 3,441 37.6 2.3 1.1 *975 48.5 -1.2 2.0 648 35.8 -2.2 2.3 Qualifications achieved by the calendar year after starting YS Level 1 + 5,709 51.7 3.2 0.8 *NA NA NA NA NA NA NA NA Level 2 + 5,709 39.9 2.0 0.8 *2,040 81.4 1.1 1.1 NA NA NA NA Level 3 + 5,709 12.7 -1.3 0.6 *2,043 43.0 -4.3 1.6 *1,071 59.1 -0.5 1.6 Receiving a benefit 6 months 5,700 9.5 1.7 0.5 *2,037 6.5 2.1 0.6 *1,068 7.5 2.4 1.1 * 12 months 5,691 19.2 3.5 0.7 *2,034 11.1 2.7 0.9 *1,068 13.4 3.7 1.4 * 18 months 5,643 26.9 3.7 0.8 *2,013 15.5 3.7 1.0 *1,053 16.2 4.3 1.4 * Other outcomes targeted by the programme Level 2 qualification by end of year when turned 18 3,348 40.6 2.5 1.1 *1,434 80.5 0.3 1.4 NA NA NA NA Benefit receipt in the 3 months after the 18th birthday 4,395 35.1 6.6 1.0 *1,788 14.4 4.4 1.1 *657 12.2 3.4 1.7 * Not in employment, education or training 6 months 5,700 26.7 -5.9 0.8 *2,037 11.2 -2.3 0.9 *1,068 15.4 -0.5 1.3 12 months 5,688 32.5 -2.7 0.8 *2,037 14.7 -0.5 1.1 1,068 18.0 0.4 1.4 18 months 3,438 37.8 -1.3 1.2 975 18.6 -0.3 1.8 651 18.3 -0.2 1.7 In employment 6 months 5,700 18.6 -3.9 0.7 *2,037 32.7 -2.0 1.4 1,068 42.2 -1.7 1.9 12 months 5,688 25.0 -3.0 0.7 *2,034 40.5 -1.5 1.4 1,068 50.7 -0.7 1.6 18 months 5,640 31.2 -1.4 0.8 2,013 50.8 0.6 1.5 1,053 56.4 -0.7 1.7 Notes: All sample size numbers are randomly rounded. Estimates that are statistically significant at the 95% confidence level are marked with an asterisk.