OPEN SCIENCE AND PSYCHOPHYSICS: TOOLS TO STUDY THE MIND

OPEN SCIENCE AND PSYCHOPHYSICS: TOOLS TO STUDY THE MIND Diana Eugenie Kornbrot School of Life and Medical Science, University of Hertfordshire [email protected] Abstract Background: psychophysics and quantitative methods There is a long honourable history of using quantitative data to study the human mind. It comprises numerical measures of performance that vary among individuals and may be affected by experience, training and motivation; and laws relating these measures to each other. Raw measures include: sensation via magnitude estimation and Likert categories; discrimination as proportions of hits and false alarms; and reaction time. Derived theoretical measures include: Steven’s power law coefficient; d’ and associated response bias criteria; information accrual rate for timed tasks with associated speed accuracy criteria. Much of experimental psychology uses statistics to estimate these descriptive measures and their variance; to test hypotheses about whether predictor effects according to task or population might have occurred by chance (p-value); and the magnitude of any predictor relative to the inherent variability of the target measures (effect size). The replicability crisis and the Open Science movement Recently many hallowed experimental results have been called into question as replications fail or are impossible due to unavailability of raw data or details of statistical methods. There is also systematic bias, as non-significant results may not be reported. Criteria for effective Open Science tools in response to the crisis are considered, including good graphics, effect sizes, and non-linear options. I have created a set of infographic tools embodying Open Science guidelines. These comprise: overview; abstract; introduction (purpose and prior knowledge); sample description (method 1); variables (method 2); analyses including code (method 3); graphic summaries (results 1); descriptive statistics (results 2); inferential statistics (results3); summary and impact; references; acknowledgements; supplementary material; pre-registration. Example use free R based package JAMOVI. G* is useful for a priori power and Lenhard’s package for interpreting effect sizes. R, which may require coding, is recommended for more complex theoretical parameter estimation, e.g., for information accrual models. AI and large language tools are available for: literature reviews, automated summaries of documents and across articles, data extraction and chat functions (Consensus, Scitex, Elicit, SciSpace). AI writing assistants may help with paraphrasing/summarising (Quillbot, Jenni). Map based systems provide cross literature links (Litmaps, Researchrabbit , Connectedpapers). How these evolving tools fare for psychophysics is explored, with some interesting and amusing results. Summary Open Science tools are a natural, if arduous, evolution from early psychophysics. In my view they are the way forward to enhance progress in scientific psychology and psychophysics. 0 Open science and psychophysics: quantitative tools to study the mind Diana E Kornbrot School of Life and Medical Sciences, University of Hertfordshire, UK [email protected] October 2024 Quantitative methods 1 Plan Quantitative methods October 2024 ➢Review history of psychophysics ❑Early attempt to quantify psychological phenomena Psychophysics, Quantitative Psychology ❑Sensation, performance, reaction time, errors ❑Models: signal detection, Information accrual, Decisions ➢Open Science ❑Movement ❑Tools ➢Artificial intelligence ❑Information communication: learning, teaching, advocacy ❑Literature review tools: references, grey literature ❑Data analysis ❑Useful prompts and strategies 2 Open Science: Movement and Practice Quantitative methods October 2024 ➢Replicability Crisis ❑Many treasure psychological and medical studies do not replicate Poor design: recruitment [too few, WEIRD] , methods and materials not specified Community discourages replication > file draw problem ➢Response ❑Guidelines: Mss., Journals, Reviewers ❑Registered reports ➢Guidelines ❑Minimal not advocacy ❑Includes further resources ❑Can be used as check list ❑Version 1: single page infographic ❑Version 2a: infographic for each stage ❑Version 2b: results examples using JAMOVI [R-based, free, click and drop interface] 3 Open Transparent Replicable Science Guide ❖Investigation Stages from Start to Finish ❖Open Internet Principles ❖Open Science Principles ➢FAIR Findable, Accessible, Interoperable, Reusable ➢Open Science Foundation UK Reproducibility Network ❖Section 1. Planning and Record Keeping ❖Section 2. Report or Ms. Production ➢Include Data Statement with doi for deposit, email for data/materials authoriser ❖Section 3. Permanent Supplement, if too long for main Ms/appendix ➢Material (stimuli, protocols…); Analysis; Code; Raw Data (numeric, qualitative); tables, figures ➢Useful depositories: Author Institution or Public, e.g. Figshare OSF NOT journal/publisher site if behind paywall, as not Accessible 0 October 2024 Quantitative methods 4 1. Planning, Recording, Execution ❏Keep good records of progress ❏As plans evolve and decision are made ❏Essential to good science ❏Data management plan from start OSF, ESRC ❏Ensure secure backup ❏Consider project management tools in R ❏File/folder names: Memorable, searchable, sortable ❏Start with date as YYYYMMDD (either start or last modified) ❏Author (if not confidential). Topic/Project; possibly subtopic ❏Folder for each project: Possibly sub-folders for each study ❏Confidential material held separately ❏All replicability information: materials, procedures, analyses ❏Project data: all versions ❏Draft Mss, blogs, announcements, summaries, etc. 1 3-Dec-20 Open Science Kornbrot [email protected] htp://dianakornbrot.wordpress.com/ @dekaliss October 2024 Quantitative methods 5 1.1 Global Resources for many projects ❖Frequently used templates ➢Journals ➢Ethics forms ➢Lab notebooks, Lab reports, etc. ❖Reference database for ALL projects ➢Zotero, Mendeleev, Endnote (expensive but can migrate) etc. ➢One database for all projects ➢Topic and Ms, specific sub-folders ❖Calendar: key dates including deadlines ❖Contacts: colleagues, journal or sponsors, etc. October 2024 Quantitative methods 6 1.1 Replicability Plans Folder ❖Memorable name + start & current date YYYY/MM/DD format, easy sort ➢Always record, as decisions are made ➢Copy/paste between items. EFFICIENT ➢Option 1: Record in completely free form logbook file ➢Option 2: Record direct into component Ms. files or tools ➢Data management, Pre-registration, Ethics, Ms. Supplement ❖Documents [one or more] for: ➢Goals for introduction, informs analysis ➢Method: Instruments, tools, variables, analyses ➢Results implications and conclusions ➢Data management plan, UKRC, EU ➢Other communication files: blog, summaries, conferences ➢Conferences, press releases, blog, social media, include doi of main report October 2024 Quantitative methods 7 1.2 Data folder Some confidential ❖Keep & backup initial and final versions, ALL files, versions ❖1.2.1 Completely Raw Data. Confidential ❑As collected. May be in special format of package ❑Video, audio, text: qualitative data. Productions: drama, art, media ❑Pictures for brain scans, location data ❖1.2.2 Universally readable numeric Raw Data ❑Text or spreadsheet. As exported from package: may include Confidential ❖1.2.3 Tidy: text/csv or spreadsheet. Confidential ➢Unneeded columns removed, interpretable column names ➢Upload to secure site, e.g. institution one drive ❖1.2.4 Shareable data for Supplement Anonymised ➢ALL shareable data. Highly desirable ➢Data supporting results/conclusions. Essential ❏Include all individual parameter estimates supporting summary Tables or Grab[h ➢Numeric data: text or spreadsheet, headed columns ➢Non-numeric data: qualitative, individual scans, arts products, etc October 2024 Quantitative methods 14 3.2 Replicability Supplement: What? ❏Materials, protocols, ethics: if too complex for main Ms. or Appendices ❏Formats include: audio, visual, media, surveys, experiment generators etc. ❏Data: May be discipline specific. e.g. human is anonymised GDPR, special for internet studies ❏Supporting ALL results required. All shareable desirable, but NOT required Quantitative numeric, Format FAIR = Accessible ❏ReadMe.txt with variable details; data set(s) comma separated values, CSV Recommended ❏Spreadsheet (e.g. EXCEL, googles sheets): ReadMe sheet + sheet for each data set. Easier to use ❏Spreadsheet details in or spreadsheet format ESSENTIAL for FAIR, Accessible ❏NEVER package format e.g. SPSS .SAV, as NOT Free. ❏NEVER unstable free software, e.g. R, Python, may not be Reusable ❏Other data: raw scans, transcripts etc. Analysis Codes and Scripts: if too complex for main Ms. or Appendices ❏Free code, e.g. R or Python. Proprietary syntax/code as text, e.g. SPSS, SAS ❏Software (e.g. C, Stan) code with version & operating system October 2024 Quantitative methods 15 Artificial Intelligence for Research Quantitative methods October 2024 ➢ChatGPT 4.0.1 general Large Language Model, LLM ❑https://chatgpt.com/ Paid version to explore ❑Invents references and other data ➢Knowledge Summary & Acquisition ❑History, dates, references, key people ❑Idea generation, brainstorming ❑Information: Psychophysics, Signal detection, Reaction time, Information accrual, decisions ❑Literature search ➢Advocacy, marketing and writing/rewriting ❑International Society for Psychophysics, Radical statistics, ❑Open Science Foundation, Campaign for Science Engineering, Scholars at risk ❑Ms. and conference abstracts ❑Research and innovation proposals ➢Other AI tools: vision, speech recognition, visual productivity 16 Artificial intelligence in Academic writing Quantitative methods Octobe 2024 ➢AI's impact Khalifa, M., & Albadawy, M. (2024). Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, 5, 100145. https://doi.org/10.1016/j.cmpbup.2024.100145 ➢Academic writing and research is transforming practices ❑Source 24 studies from major databases since 2019 highlights ❑Necessitating broader integration and ethical use in research ➢AI enhances academic writing in six areas: ❑idea generation; content structuring; literature synthesis ❑data management ; editing; ethical compliance ➢ChatGPT benefits ❑helps manage complex ideas and extensive information ❑demonstrates significant potential in academic writing ➢ChatGPT challenges ❑academic integrity ❑AI-human balance ➢•Future recommendations for ongoing AI research in academia ❑Training, ethical usage, and transparent integration in workflows. 17 Useful Prompts Quantitative methods October 2024 ➢Audience ❑Academic: student: UG, PG, research, conference promotion, society promotion ❑Lay: most important results, market ➢Format ❑Table with specified columns, format suitable for copying ❑Reference list with doi, format for reference manager (zotero, mendelev, endnote) ❑Save dialogue as word file. Do it OFTEN. Specify ALL words from you and assistant Searching chatgpt output is a NIGTHMARE ➢Detail ❑Start general ❑Refine by additions, and breakdown ➢Type ❑Summary, CV, job application ➢Advice? 18 Idea generation Quantitative methods October 2024 URL Score Source Score Source jasper.ai 4.7/5 Trustpilot chatgpt 2.3/5 chatgpt 4.6/5 OpenAI Community sudowrite.com 4.5/5 Capterra notion.so 4.4/5 G2 rytr.me 4.4/5 Trustpilot inkforall.com 4.3/5 Product Hunt wordtune.com 4.2/5 Capterra scite.ai 4.2/5 ResearchGate 19 Knowledge Acquisition ➢Aim: What might general public learn of our discipline? ➢Request to chatgpt “History psychophysics & quantitative psychology” ❑Psychophysics detail from dialogue or my knowledge Models, reaction time, information accrual signal detection, binary decisions Names I got 2 imaginary references ❑Quantitative Intelligence ➢Measurement in psychology ❑variable type no mention LIkert ❑analysis no mention generalized linear models, effect sizes ANOVA, non-parametrics. As old-fashioned text ➢Generally poor ❑Need good secondary reference or text. AI may or may not find October 2024 Quantitative methods 20 Knowledge Acquisition Limitations ➢Key topics omitted ❑DDM direct diffusion model, not LBA linear ballistic accumulator ➢Invented references ❑Also invented DOI, could not perform check of existence ➢Misattribution ❑Steve Link LBA Eventually dissuaded with “exact text for Steve Link and LBA” Told off chatgpt. Got apology Next time as chatgpt remembers previous dialogue: Got LBA names correct. Steve detached from LBA, but still insisted his work was precursor! ➢Traditional sources better for literature search ❑But general search helps with search keywords ❑Web of Science Scopus, GoogleScholar October 2024 Quantitative methods 21 Chatgpt missed ➢Several early psychophysicists showed an interest in parapsychology, as it was closely related to investigations of human perception and consciousness. A few notable figures include: 1. Gustav Fechner –Known as the father of psychophysics, Fechner also had a strong interest in spiritualism and parapsychology. His work in measuring the relationship between stimuli and sensations extended into his metaphysical beliefs, where he explored concepts of the soul and the afterlife. Sent from my iPhone October 2024 Quantitative methods 22 Moving ON: some examples ➢Rewriting ❑Abstracts ❑Book plugs: Maverick on my WordPress Amazon ❑Society information Please rewrite this to attract new members, followed by above text ➢Attracting students to psychology by country and level ➢Literature Reviews October 2024 Quantitative methods 23 ChatGpt abstract this presentation: Psychophysics, Quantitative Methods, and Open Science Quantitative methods October 2024 ➢Psychophysics has a long tradition of using quantitative data to understand the human mind, focusing on measurable variables like sensation, discrimination, and reaction time. Key theoretical measures include Steven’s power law coefficient, signal detection theory metrics (d’), and information accrual rates in timed tasks. Statistical methods are widely used to estimate these measures, test hypotheses, and assess the impact of various predictors. ➢The replication crisis and the Open Science movement have raised concerns about the reliability of past findings, leading to demands for more transparency and accessibility in research. Effective Open Science tools should include clear visualizations, effect sizes, and non-linear modelling options. Various platforms (e.g., JAMOVI, G*, R) can aid in this process. ➢AI and large language tools are becoming increasingly useful in research. Tools like Consensus, Scitex, Elicit, and SciSpace can perform literature reviews, automate document summaries, and extract data. Writing assistants like Quillbot and Jenni help with paraphrasing and summarizing, while map-based systems like Litmaps, Researchrabbit, and Connectedpapers assist in linking research across studies. These tools are still evolving but are showing promising applications in psychophysics. ➢Open Science is seen as a challenging but essential progression from the early days of psychophysics, promising to improve the rigor and reproducibility of psychological research. 30 Open Science Foundation NOW ➢https://osf.io/ Home. Scroll for content to ❑As a collaboration tool OSF helps research teams work on projects privately or make the entire project publicly accessible for broad dissemination. As a workflow system, OSF enables connections to the many products researchers already use, streamlining their process and increasing efficiency. ❑Structured projects Manage files, data, code, and protocols in one centralized location and easily build custom organization for your project —no more trawling emails to find files or scrambling to recover lost data. ❑Controlled access Manage which parts of a project are public or private, making it easy to collaborate and share with the community or just your team. ❑Enhanced workflow Connect your favourite third-party services like Dropbox or Google Drive, automate version control, preregister your research, share preprints. ❑Dependable repository OSF's preservation fund is sufficient for 50+ years of read access hosting at present costs. October 2024 Quantitative methods 31 Radical Statistics 1 Quantitative methods October 2024 ➢About ❑The Radical Statistics Group, usually abbreviated to Radstats, was formed in 1975 as part of the radical science movement associated with the establishment of the British Society for Social Responsibility in Science (BSSRS). ❑The researchers and statisticians who started Radstats shared a common concern about the political implications of their work and an awareness of the actual and potential misuse of statistics. Sadly, neither the BSSRS nor other organizations affiliated to the BSSRS, survived the 18 years of Conservative rule from 1979 to 1997. ❑RadStats is now independent of any other organisation. Members are ‘radical’ in being committed to helping build a more free, democratic and egalitarian society. Members of Radstats are concerned at the extent to which official statistics reflect governmental rather than social purposes. ➢Chat ❑The Radical Statistics Group, or Radstats, was founded in 1975 as part of the movement promoting socially responsible science, including the British Society for Social Responsibility in Science (BSSRS). The founders of Radstats were dedicated to examining the role of statistics in society, focusing on how they can be used to support positive social change. Though originally linked to BSSRS, Radstats has evolved into an independent organization committed to fostering a more free, democratic, and egalitarian society. ❑The group’s key mission is to ensure that statistics are used ethically and transparently to serve social interests. Members of Radstats are devoted to addressing how official statistics are produced, with an emphasis on aligning them with societal, rather than purely governmental, objectives. 32 Open Science Foundation 1: chatgpt ➢The Open Science Framework (OSF) is a platform designed to support researchers in managing and sharing their work efficiently. Here is a summary of its key features: ❑As a Collaboration Tool ❑Project Management OSF allows teams to manage their research projects either privately or publicly, making the sharing and dissemination of work straightforward. ❑Workflow Integration: It enables seamless integration with many commonly used research tools, increasing efficiency and simplifying workflows. ❑Structured Projects Centralized Organization: Users can organize all files, data, code, and protocols in one location, reducing the need to search through multiple sources for project information. ❑Custom Structure: Custom organizational structures can be set up for each project, improving data management and access. October 2024 Quantitative methods 33 Open Science Foundation chatGPT 0 ➢https://osf.io/ Home. Scroll for content. Text modified with heading ➢As a collaboration tool, OSF Helps research teams work on projects privately or make the entire project publicly accessible for broad dissemination. As a workflow system, OSF enables connections to the many products researchers already use, streamlining their process and increasing efficiency. ➢Structured projects ❑Manage file data, code, and protocols in one centralized location and easily build custom organization for your project —no more trawling emails to find files or scrambling to recover lost data. ➢Controlled access ❑Manage which parts of a project are public or private, making it easy to collaborate and share with the community or just your team. ➢Enhanced workflow ➢Connect your favourite third-party services ❑Dropbox or Google Drive, ❑Automate version control, preregister your research, share preprints. ➢Dependable repository ➢OSF's preservation fund ❑Sufficient for 50+ years of read access hosting at present costs. October 2024 Quantitative methods 34 Open Science Foundation 1: chatgpt ➢The Open Science Framework (OSF) is a platform designed to support researchers in managing and sharing their work efficiently. Here is a summary of its key features: ❑As a Collaboration Tool: ❑Project Management OSF allows teams to manage their research projects either privately or publicly, making the sharing and dissemination of work straightforward. -Workflow Integration: It enables seamless integration with many commonly used research tools, increasing efficiency and simplifying workflows. ❑Structured Projects Centralized Organization: Users can organize all files, data, code, and protocols in one location, reducing the need to search through multiple sources for project information. Custom Structure: Custom organizational structures can be set up for each project, improving data management and access. ❑Controlled Access Privacy Settings: Users have control over which parts of their projects are visible to the public and which remain private, allowing flexibility in collaboration within the team or broader community. ❑Enhanced Workflow** Third-Party Integration: The platform supports connections to external services like Dropbox, Google Drive, and more, helping toautomate workflows, manage version control, preregister research, and share preprints. ❑Dependable Repository** Long-Term Preservation: OSF’s preservation fund ensures that research data and files are hosted with reliable read access for 50+ years at current costs, making it a stable repository for long-term access. ➢Overall, OSF is designed to streamline research project management, enhance collaboration, and provide dependable data storage. October 2024 Quantitative methods 35 Open Science Foundation chatGPT 1 ➢Join the Open Science Framework (OSF) Community Today! ➢Are you looking to elevate your research collaboration and streamline your workflow? The Open Science Framework (OSF) is here to transform how you manage, share, and discover research. ➢Why Choose OSF? ➢Seamless Collaboration ❑Private and Public Projects ❑Collaborate effortlessly with your team in a secure, private space, ❑Share your entire project publicly to reach a broader audience and enhance the impact of your work. ➢Centralized Management ❑Keep all your files, data, code, and protocols in one organized location. ❑Say goodbye to endless email threads and the frustration of lost data. ➢Enhanced Workflow Integration ❑Connect Your Favourite Tools: ❑Integrate OSF with the research tools you already use Dropbox, Google Drive, and more. October 2024 Quantitative methods 36 Open Science Foundation chatGPT 2 ➢Automated Version Control: ❑Never lose track of changes. OSF automatically records revisions, Ensure you always have access to previous versions of your work. ➢Preregistration and Preprints: ❑Easily preregister your studies ❑Share preprints to promote transparency and accelerate the dissemination of your findings. ➢Flexible Access Control. Customizable Privacy Settings: ❑You decide which parts of your project are public or private. ❑Collaborate privately with your team or ❑Share your breakthroughs with the global research community. ➢Community Engagement: ❑Engage with other researchers, discover new collaborations ❑Contribute to a vibrant, open scientific ecosystem. ➢Dependable and Long-Term Storage ❑A preservation fund ensuring 50+ years of read-access hosting at current costs, ❑Your research is safeguarded for the future. October 2024 Quantitative methods 37 Scholars at Risk NOW ➢About ❑Scholars at Risk is an international network of institutions and individuals whose mission it is to protect scholars and promote academic freedom. By arranging temporary academic positions at member universities and colleges, Scholars at Risk offers safety to scholars facing grave threats, so scholars’ ideas are not lost and they can keep working until conditions improve and they are able to return to their home countries. ❑Scholars at Risk also provides advisory services for scholars and hosts, campaigns for scholars who are imprisoned or silenced in their home countries, monitoring of attacks on higher education communities worldwide, and leadership in deploying new tools and strategies for promoting academic freedom and improving respect for university values everywhere. ➢ChatGPT ❑Scholars at Risk is an international network dedicated to protecting scholars and promoting academic freedom. It helps scholars facing serious threats by arranging temporary academic positions at partner institutions, allowing them to continue their work safely until they can return to their home countries. The organization also provides advisory services, advocates for scholars who are imprisoned or silenced, monitors global attacks on higher education, and develops new strategies to enhance academic freedom and respect for university values. ➢Omits individuals and institutions October 2024 Quantitative methods 38 Campaign for Science and Engineering Quantitative methods October 2024 ➢Home ❑The Campaign for Science and Engineering (CaSE) is the UK’s leading independent advocate for science and engineering. We represent over 115 scientific organisations including businesses, universities, professional bodies, and research charities as well as individual scientists and engineers. Collectively our members employ over 336,000 people in the UK, and our industry and charity members invest around £32.2bn a year globally in R&D. ➢Chat 1 ❑The Campaign for Science and Engineering (CaSE) is the UK’s leading independent advocate for science and engineering, playing a crucial role in shaping the nation's future. We represent over 115 scientific organizations— spanning businesses, universities, professional bodies, and research charities—alongside individual scientists and engineers. Together, our collective expertise and advocacy influence policies and initiatives that drive progress across all sectors of science and engineering. ➢Note chat lost number employed and £ spent ➢Chat Retry ❑The Campaign for Science and Engineering (CaSE) is the UK’s leading independent advocate for science and engineering, representing over 115 scientific organizations, including businesses, universities, professional bodies, and research charities, as well as individual scientists and engineers. Collectively, our member organizations employ over 1 million people in the UK, contributing significantly to the economy and society. In addition, our industry and charity members invest around £32.2 billion a year globally in research and development, driving innovation and technological progress across various sectors. 39 Radical Statistics 2 Quantitative methods October 2024 ➢About Concerns ❑The mystifying use of technical language todisguise social problems as technical ones ❑The lack of control by the community over the aims of statistical investigations, the way these are conducted and the use of the information produced ❑The power structures within which statistical and research workers are employed and which control the work and how it is used ❑The fragmentation of social problems into specialist fields, obscuring connectedness ➢Chat Our core areas of focus include: ❑Ensuring that statistical language is clear and transparent, so social issues are not disguised as mere technical challenges. ❑Empowering communities to have greater input and oversight over the aims, methodologies, and uses of statistical investigations, ensuring that data serves the interests of society as a whole. ❑Addressing the influence of power structures in the employment of statistical and research professionals, advocating for work that is free from undue control and can be used for the public good. ❑Promoting a holistic approach to social issues by preventing their separation into overly specialized fields, which can obscure their interconnectedness and hinder comprehensive solutions. ➢Chat is worse. Longer more buzz word –e.g. holistic, empowering