WoRXR

5th Workshop on Replication in Extended Reality (WoRXR) Program •14:00 - 14:15 Welcome •14:15 - 15:00 Ensuring Replicability: Data Storage & Digital Appendices. Tutorial by Florian Weidner •15:00 - 15:30: Coffee Break •15:30 - 16:00: Keynote - Beyond Initial Findings: Replication, Reliability, and Statistical Practices in Empirical Studies by Daniel Zielasko •16:15 - 16:30: Perception Thresholds for Unobtrusive Movement in VR Versus AR by Verena Biener •16:30 - 17:00: Brainstorming and agenda setting, wrap-up Why? •Failure to replicate many published findings, even textbook findings •Research biases •Publication bias: only significant results get published •Selection bias: only significant results analysed •Reporting bias: only significant results reported in paper •Replications? •Rarely funded •Rarely published →No incentive; most research exploratory … at IEEE ISMAR and IEEE VR? Mohammed Safayet Arefin, Verena Biener, Nijhum Hasan S M Rashidul, Florian Weidner, Jens Grubert, and J. Edward Swan. 2025. The State of Replication at IEEE ISMAR and IEEE VR: A Scoping Literature Review (2010 –2024) and Online Survey. In 2025 IEEE International Symposium on Mixed and Augmented Reality, 2025. •14:00 - 14:15 Welcome •14:15 - 15:00 Ensuring Replicability: Data Storage & Digital Appendices. Tutorial by Florian Weidner •15:00 - 15:30: Coffee Break •15:30 - 16:00: Keynote - Beyond Initial Findings: Replication, Reliability, and Statistical Practices in Empirical Studies by Daniel Zielasko •16:15 - 16:30: Perception Thresholds for Unobtrusive Movement in VR Versus AR by Verena Biener •16:30 - 17:00: Brainstorming and agenda setting, wrap-up Replicability and data/algorithm storage and availability Florian Weidner, University of Glasgow Outline for the next 30-45 minutes •Why publish data and algorithms? •What could (should) we store? •Where to store? •Zenodo •OSF •ACM and IEEE •How do we document it? •Summary Why publish data? Reproducibility and replicability. For example: •Prototype shared: test exactly the same. •Data shared: calculate exactly the same •Everything: do exactly the same. But also: •Comparison with new findings easier (statistics) •Making improvements is easier (iterations) Reproducibility: reanalysing existing data in the same manner. Replicability: reconducting the analysis, including data collection. … do study … OSF Workflow Add data. OSF comments •Free •Foundation well-funded (long-term perspective) •Preregistration •Anonymous links •Versioning possible •50GB •Links to github release →no file mirror. •Covers research lifecycle https://github.com/eblondel/zen4R Tl;dr: OSF 1. Create and fill out pre-registration. 2. Do study. 3. Upload data, analytics code, materials, papers, and supplemental. 4. Create view-only links (private or not) 5. Add them to manuscript. 6. Be happy ☺ Example: IEEE VR 24 submission: https://osf.io/uf46b/ Zenodo All about storage. Zenodo UI •Signup / login •Create new upload •Get DOI •Fill in metadata •Upload files •Review •Publish ➔straightfoward Example Private? 1. Create anon repository 2. Publish restricted 3. Create link 4. Share link (change authors later) Comments on Zenodo •Free •Funded by CERN (long-term perspective) •Versioning possible •50GB •Links to github release •auto-update on new release •Nice github Zenodo badge https://github.com/eblondel/zen4R ACM / PCS / TAPS and IEEE Summary All together? •OSF: allrounder, free, pre-registration, 50GB = more work? •Zenodo: Github, free, 50GB, no pre-registration •ACM / PCS / TAPS: Free, limited file size undefined undefined What now? •Make materials available. My (soft) requirements •Stripped executable + setup instructions •Study protocol and self-defined questions •Raw data (if possible), aggregate data + scripts to calculate results •What we should do: OSF with pre-reg (time, though) •What I actually do: •Submission: •Upload most important stuff to PCS •Reserve and link Zenodo/DOI in paper in acknowledgements (anon) •After acceptance •Data, scripts, prototype, paper, and video on Zenodo •Link/DOI in paper For aspiring academics … it makes a nice part of your academic cv for applications Thank you! Questions?