Blinded Assessment of Study Results From a Randomized Trial on Policy Experiment Aversion

Blinded Assessment of Study Results From a Randomized Trial on Policy Experiment Aversion Discussion meeting Oct 15, 2025 Attendees: Atle Fretheim, Mona Bjørbæk, Ingeborg Hess Elgersma, Runar Barstad Solberg, Petter Elstrøm (online), Evy Therese Dvergsdal Furøy (online), Christine Holst, Sarah Ellen Rosenbaum, Christopher James Rose, Erle Refsum (minutes, corresponding author: [email protected]) The protocol and statistical analysis plan have previously been published (Elgersma et al., Policy Experiment Aversion: Protocol and Analysis Plan for a Survey and Embedded Randomized Trial Conducted During a National Debate About a Real-Life Policy Experiment on the Effectiveness of an In-Work Tax Credit (August 27, 2025). Available at SSRN: https://ssrn.com/abstract=5408584 and http://dx.doi.org/10.2139/ssrn.5408584). IHE performed the analyses and was not blinded to treatment allocation. The other meeting attendees were blinded to treatment allocation. IHE presented blinded results, labelling the treatment arms “Swifties” and “Non-Swifites”. IHE was present but did not comment on the research group’s interpretations of the results and was not involved in the writing of this document. Tables with unweighted and weighted proportions and weighted complete-case effect estimates were presented for the primary outcome (policy experiment aversion), secondary outcomes (differential treatment justifiability and inferred experiment aversion), and subgroup analyses. Though exact numbers and proportions were presented at the meeting, only effect estimates are referred here to maintain confidentiality. The intervention consisted of a brief explanatory text on randomized trials in the context of policy making, see protocol for details. Of the 4,323 individuals invited to participate, 1,241 (28.7%) enrolled in the study. Among these, 1,225 (98.7%) were included in the weighted complete-case analysis. Multiple imputation was not used to account for the missing data, as the response rate exceeded the prespecified threshold of 95%. Interpretation of study results if “Non-Swifties” received the intervention The effect estimate of the primary outcome -experiment aversionis a risk ratio (RR) of 2.08 (95%CI 1.40 to3.09) comparing “Swifties” to “Non-Swifties” (reference). Assuming that “NonSwifties” received the intervention, the analysis of the primary outcome suggest that the intervention reduces experiment aversion by a factor of about two. This finding aligns with the a priori hypothesis that an explanatory text would reduce skepticism toward policy experiments. The effect size estimate is substantially larger than we had expected. Analyses of the secondary outcomes align with the primary outcome: With an effect estimate of RR 0.68 (95%CI 0.53-0.88), we infer that the “Non-Swifties” treatment – presumed here to be the intervention - increased acceptance of differential treatment. The analysis of inferred experiment aversion accounted for a potential floor effect, whereby participants who disagree with experimenting to a similar degree as implementing or not implementing are counted as experiment averse. This result also aligns with the analysis of the primary outcome, indicating that the “Non-Swifties” treatment – also presumed here to be the intervention - decreases experiment aversion with a RR 1.59 (95%CI 1.15-2.20) when a floor effect is considered. However, the effect was less pronounced than in the primary analysis. Interpretation of study results if “Swifties” received the intervention Assuming the intervention was applied to the “Swifties” group, the estimated RR of 2.08 (95%CI 1.40-3.09) comparing “Swifties” to “Non-Swifties” (reference), we infer from analysis of the primary outcome that the intervention increased aversion to policy experiments. In this case, the effect of an explanatory text does not align with the a priori hypothesis. The finding that increased information has a seemingly large, paradoxical effect should be explored further, as it may have severe consequences on how the NIPH communicates with the public. Analyses of the secondary outcomes align with analysis of the primary outcome: The intervention leads to decreased acceptance of differential treatment, with a RR of 0.68 (95%CI 0.53-0.88) and effect of the intervention is sustained also when a floor effect is considered and the inferred experiment averse are included in the analysis, RR 1.59 (95%CI 1.15-2.20). Subgroup analyses for primary outcome In general, the subgroup analyses are limited by a lack of statistical power and should be interpreted with caution. Based on the effect estimates, the results indicate a trend with a higher effect on increasing age, although with no statistical significant differences between the age groups: in the youngest age group we estimate the effect to be RR 1.4 (95%CI 0.12-16.97), with a gradual increase towards the eldest age group, with a RR of 3.52 (95%CI 1.25-9.95). The p-value for interaction is 0.149, which we interpret to mean that there is not statistically significant evidence of effect modification by age. Similarly, there is no statistically significant evidence of effect modification by gender: RR for women = 2.21 (95%CI 1.27-3.86) and men = RR 1.98 (95%CI 1.12-3.49), p-value for interaction=0.749. The sample size for the “Other/Prefer not to say” group is too small to facilitate meaningful statistical analysis. With respect to education level (low, middle, high), there appeared to be statistically significant evidence of effect modification (p-value for interaction=0.028). However, it is likely that this is driven by the effect estimated for the low education level group, which was excessively large due to the very small sample sizes in this subgroup. We agreed that the p-value for interaction should be computed to test the hypothesis of equality of effect for the middle and high education subgroups (i.e., the low education level subgroup should be omitted from this analysis). The RR for middle education = 1.11 (95%CI 0.59-2.09) and high education = RR 1.86 (95%CI 1.38-2.51). If the interaction with education level persists when only including middle and high education levels, this could imply that there is an increased effect among participants with higher education levels. This again might imply that participants with higher levels of education are more likely to understand, accept or trust the explanatory text on policy experiments. There is no statistically significant evidence of effect modification by voting intention (p-value for interaction=0.62). The effect estimates vary from an RR of 2.0 (95%CI 0.87-4.61) for intended voters of the Labour Party to an RR of 2.82 (95%CI 0.84-9.39) for intended voters of the Progress Party. There is no statistically significant evidence of effect modification by awareness of (i.e., having heard about) the proposed policy experiment (p-value for interaction=0.91). We find similar effect estimates comparing participants who had heard of the proposal and not, with RRs of 2.11 (95%CI 1.35-3.29) and RR 1.93 (95%CI 0.89-4.2), respectively.