Switching a cross-national face-to-face panel survey to alternative data collection modes: Challenges and insights from the Survey of Health, Ageing and Retirement in Europe (SHARE)

Bergmann, Michael

doi:10.5281/zenodo.17710114

Switching a cross-national face-to-face panel survey to alternative data collection modes: Challenges and insights from the Survey of Health, Ageing and Retirement in Europe (SHARE)

Bergmann, Michael

Publisher: Zenodo
DOI: 10.5281/zenodo.17710114
Repository: zenodo.org
Language: en

Abstract

The Survey of Health, Ageing and Retirement in Europe (SHARE), like many other surveys, put huge efforts into the realization of an additional COVID-19 survey including a mode switch to telephone interviewing. While this dataset is interesting on its own, it is even more so with regard to its utilization in combination with the wealth of existing panel data from more than 400,000 face-to-face interviews from eight waves of data collection, including life histories of respondents. At the same time, the changes to mode and timing of data collection raise a number of important questions that have not been sufficiently answered yet. Against this background, we want explore the effects of a mode switch during an ongoing panel survey and in particular for SHARE’s specific target population of people aged 50 years and older. We are primarily interested in two connected research questions: (1) Who does and who does not respond in a certain interview mode (mode selection effect)? (2) How is a question answered in a certain interview mode (mode measurement effect)? To answer these questions, we focus on data from the eighth wave of SHARE until its suspension in March 2020 and the two waves of the SHARE Corona Telephone Survey (SCS) fielded in summers 2020 and 2021, yielding about 100,000 additional interviews. Based on these data, we analyze potential effects of the introduced changes in data collection with a particular focus on the older population and give practical advice for researchers facing similar challenges in directing their efforts.

Full text

Switching a cross-national face-to-face panel survey to alternative data collection modes: Challenges and insights from the Survey of Health, Ageing and Retirement in Europe (SHARE) Michael Bergmann Dublin June 12th 2025 1. Impact of mode switch (CAPI →CATI) during the Covid-19 pandemic a. participation b. data quality (representativeness, comparability) c. survey costs 2. Lessons learned & outlook a. SHARE Self-completion Questionnaire (SCQ) Content Michael Bergmann | 13 June 2025 Participation – 1st SHARE Corona Survey Michael Bergmann | 13 June 2025 Individual response/retention rates (AAPOR, RR4; excl. households without valid telephone numbers) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Participation – 2nd SHARE Corona Survey Michael Bergmann | 13 June 2025 Individual response/retention rates (AAPOR, RR4; excl. households without valid telephone numbers) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Representativeness: Who does/does not respond in a certain mode? Michael Bergmann | 13 June 2025 AMEs for participation in SCS1 •A couple of significant differences in the probability to participate in the telephone survey (socio-economic characteristics, health-related factors, difficulties during the interview) •Overall, moderate differences (mostly below 5%-points) •Small proportion of explained variance (R²=5.7%; without country dummies: R²=2.2%) Representativeness: Who does/does not respond in a certain mode? Michael Bergmann | 13 June 2025 AMEs for participation in SCS2 •Very similar picture (age differences are a bit more pronounced) •Overall, still moderate differences (mostly below 5%-points) •Small proportion of explained variance (R²=5.9%; without country dummies: R²=2.3%) Comparability of Data: How is a question answered in a certain mode? Michael Bergmann | 13 June 2025 Tolstrup Wester et al. (2023). Longitudinal changes in mental health following the first COVID-19 lockdown in Europe. https://doi.org/10.1515/9783111135908-008 Mode effects or true changes between Wave 8 (CAPI) and SCS1 (CATI)? Comparability of Data: How is a question answered in a certain mode? Michael Bergmann | 13 June 2025 Difference between CAPI and CATI (AMEs in %-points) Wave 8 CAPI (2019/20) SCS1 CATI (2020) Wave 9 CAPI (2021/22) Household income (<750 €) 11.0 11.9 11.6 Household income (>=4000 €) 12.0 7.7*** 10.7** Household income (don’t know & refusal) 16.0 22.3*** 19.4*** Household income (don’t know) 9.9 6.8*** 11.2** Household income (refusal) 7.3 17.4*** 10.1*** •Underrepresentation of households with high income in CATI •Higher item non-response in CATI, especially for refusals Data: SHARE Waves 8 and 9 (CAPI) and SHARE COVID-19 Survey 1 (CATI), release 9.0.0 (n=122,630). Note: Household incomes are ppp-adjusted to allow comparability across countries. Regressions include controls for differences in sample composition: age categories (60-69, 70-79, 80+; reference: 50-59), gender, educational level (ISCED level 3+4, ISCED level 5+6; reference: ISCED level 0-2), living alone, living in an urban area, born abroad, and country dummies. Results are weighted; significance level: ***: p<.001, **: p<.01, *: p<.05. Disclaimer: Survey costs are difficult to compare, but even more so here due to the change/amendment of contracts with already procured survey agencies during Covid-19! Survey Costs Michael Bergmann | 13 June 2025 Avg. total costs per interview (in €) Share of total costs per interview (CATI/CAPI) CAPI Wave 8 124 € CAPI Wave 9 120 € CATI SCS1 65 € 53% CATI SCS2 61 € 51% •At first sight CATI seems much cheaper than CAPI •However, it has to be considered that interview duration was much shorter for CATI than for CAPI (~20-25min vs. 70-80min) •Huge variation across countries (26-77% of CAPI costs) Note: Costs are purchasing power parities (PPP) adjusted. Survey costs for 22 SHARE-ERIC countries (based on procurement)