Illustrating Impacts of User Experience Work in Research Software Engineering

Illustrating Impacts of User Experience Work in Research Software Engineering Version 1.0, Dec 2025 Editors Johanna Cohoon Drew Paine Contributors Kate Arneson David Bernholdt Johanna Cohoon Rajshree Deshmukh Meag Doherty Eric Eide Eriol Fox Leigh Fu Mary J. Goldman Annette Greiner Dan Gunter Robert Ricci Drew Paine Sarah Poon Jason Wiese A Note on Contributing This white paper is a living document that we seek to add to and improve through contributions from the scientific community. If you have feedback or a vignette of your own to share, please reach out to the US-RSE UX working group on Slack. Vignettes should have a brief title, provide context for your UX work (e.g., describe an issue), explain what work was done, and explain the impact. We further direct readers to the User Experience in Scientific Software community on Zenodo, where we are gathering additional resources on UX for research software like talks, blogs, and pre-prints: https://zenodo.org/communities/uxinscience. Illustrating Impacts of User Experience Work in Research Software Engineering 1 Version 1.0 1 Editors 1 Contributors 1 A Note on Contributing 2 Overview 2 Vignettes 3 Vignette 1: Integrating essential features 4 Vignette 2: Informing policies and reducing confusion 4 Vignette 3: Improving workflows 5 Vignette 4: Prioritizing feature development 5 Vignette 5: Ensuring speed in records review 5 Vignette 6: Determining requirements and getting early wins 6 Vignette 7: Improving navigability 6 Vignette 8: Prototyping to support data management 7 Vignette 9: Improving researcher onboarding 7 Vignette 10: Simplifying scientific work 8 Vignette 11: Redesigning a core feature 8 Vignette 12: Flexibly incorporating UX to shape a product space 9 Overview Scientific progress requires reliable, usable software and computing ecosystems to support discovery rapidly and at scale. Software and scientific landscapes are also constantly evolving, introducing new requirements even for stable user bases and use cases. Additionally, research software is often developed for highly specific purposes and may struggle to gain footing among a new user group with different and complex use cases. Yet, using these software systems can often be difficult and confusing. Design decisions are often idiosyncratic and as tools undergo iterations their usability can suffer. User experience (UX) encompasses a set of techniques for researching how stakeholders interact with and interpret technology to design and implement improvements. These methods illuminate user requirements, challenges, and opportunities. With UX approaches and insights, research software engineering teams can make more informed decisions about how to spend resources and implement their visions to address user needs. UX work is fundamental to addressing the above issues of evolving requirements, complex systems, and poor usability. By applying UX methods, scientific software teams can make the practice of research simpler, more efficient, and more enjoyable. This white paper presents a series of vignettes that describe UX work completed in varied scientific contexts. They were gathered from stakeholders across the scientific community with the intent of demonstrating the value of UX work for science. Among them are stories about software products meant to serve broad populations while others address niche research communities. Regardless of context, these short reflections illustrate an array of the varied impacts and benefits that UX work can have for research software from less confusion and reduced complaints from end users to more efficient workflows and integration of in-demand features. These vignettes illustrate how UX work can deliver outsized returns with modest investments of time and resources. Teams that implement even basic UX methods quickly recognize the value and expand their user-centered approach. This is true of the people described in the vignettes below and their collaborators; initial successes with UX methods have often catalyzed organizational shifts toward user-centered design. We present these narratives as a call to action: realize the benefits of UX work in science for yourself and create usable research software. We hope that this advocacy enables more engagement and investment in these efforts. Vignettes We present a series of brief stories, sometimes anonymous, across a variety of institutions and practitioners describing how UX has been applied in research software engineering contexts. These narratives demonstrate how UX methods can meaningfully accelerate and improve scientific work. Vignette 1: Integrating essential features Our UI and experience (UIX) team works closely in collaboration with researchers, domain scientists, end users, and software engineers to develop scientific research software, among other applications. One such example is the ReactionMiner project. This chemistry-specific literature mining and search tool is a part of AlphaSynthesis—an open-source software suite within the Molecule Maker Lab Institute (MMLI) that supports a range of chemistry applications, from synthesis planning to closed-loop discovery. When we joined the project, initial user discovery sessions and usability testing revealed critical pain points in the existing ReactionMiner technology. We documented key requirements and user workflows from these sessions, such as the need for the analysis of reaction conditions across multiple documents and support for more familiar structural representations. We brought these findings back to the research team as opportunities for improvement and iteration in a longer-term product roadmap. Over several months of collaboration, this led to the integration of essential features like multi-document analysis and structural visualizations, shaping a more usable and interpretable tool. By leveraging user sessions in close collaboration with the research team, we aligned user needs with technical feasibility, enabling strategic planning for more impactful, sustainable software development. Vignette 2: Informing policies and reducing confusion Our team builds a cloud computing testbed for networking researchers. Because platform policy allowed people working together on projects to have access to each others’ node reservations, sometimes people would accidentally use reserved nodes that their collaborators were expecting to leverage themselves. Sometimes these mistakes led to missed conference deadlines. Our developers were frequently responding to questions in our help forum from users who thought their reserved nodes had gone missing. By mining our forum history and conducting user interviews, we learned about users’ confusion and devised a solution to reduce accidental use of others’ reserved nodes—we would add more information to the UI to inform users about node reservations and who was using which resources. We then surveyed users to get feedback on our UI changes and found the solution wasn’t as effective as we had hoped. So, we pivoted to a new tactic, deciding to change our policies to better match user behavior. Our policy change meant that collaborators no longer had to share access to reserved nodes, matching their expectations. We had been getting complaints because of the confusing policy for years but once we started addressing it through user research, we made the policy change within the year. The knowledge we gained of our users’ expectations and the negative effects they were experiencing was informative and motivating for our policy changes. Vignette 3: Improving workflows We aimed to create a community repository for users to find and share models and data, facilitating interdisciplinary collaboration among scientists from different fields. Although users were enthusiastic about the repository's potential, they struggled with the data-uploading process. To clarify and streamline this process, we employed UX research techniques, including stakeholder analysis, user journeys, persona development, and heuristic evaluation. Our approach allowed us to identify critical users and their key concerns regarding the process for uploading data. As a consequence of this work, the project adopted a new UI framework that improved accessibility and we streamlined the data upload process, making contributions to the repository easier and faster. Vignette 4: Prioritizing feature development Our project involved developing a UI for chemical process engineering so that users can visualize and verify models created in Python code. Our team applied UX methods, starting with a meeting where stakeholders brainstormed and prioritized features using sticky notes. The team then conducted virtual interviews with users to gather additional information. This process helped the team prioritize features and focus on the most critical aspects of the UI, resulting in a UI that better suited user needs compared to the prior UI. The UX process had a significant impact, teaching the team the importance of prioritization and helping them create a more effective and user-friendly UI. The development team appreciated the value of the stakeholder interviews, and began to use them for future projects such as when developing an API. Vignette 5: Ensuring speed in records review For many state agencies concerned with oil and gas regulation, staff have to manually look through hundreds of scanned oil well records for relevant information and then enter that information into a spreadsheet. It is time-consuming and labor-intensive for them, but commercial solutions are unaffordable. Our team sought to accelerate this work for them. We conducted structured interviews and discussions with stakeholders to understand their goals, contexts, and current workflows when using oil well records. These interviews with different state agencies and organizational staff were essential in identifying the primary and secondary users for our new tool, thus helping to prioritize the core capabilities. We found that the user base comprises a broad spectrum of professionals, including project managers, engineers, data coordinators, student interns, and scientists, each contributing to the creation, management, and utilization of these documents. Understanding the stakeholders, their roles, responsibilities and pain points helped the team to make informed decisions about the capabilities and constraints to be considered in the new tool we were building. Participants in the meeting were also shown concept mockups of the proposed tool (OGRRE) and were asked about desired features and capabilities for this tool. The feedback from these discussions was used to inform the design of the tool's workflow, which was reviewed and refined with internal stakeholders to ensure consensus and alignment. Ultimately, we developed the Oil and Gas Regulatory Record digitizEr (OGRRE), a custom user interface that facilitates the digitization, review, and extraction of structured data from scanned documents of historic oil and gas regulatory records. By leveraging AI/ML optical character recognition (OCR) and an interface tailored to our stakeholders’ needs, OGRRE accelerates the processing of historic oil well records. Vignette 6: Determining requirements and getting early wins Our project is in the early stages of development to create data management resources for a scientific community. The project will provide tools that support tasks across the data lifecycle, simplifying work and increasing data sharing across sub-disciplines. We are using UX methods to ensure that the features and capabilities we offer will meet the needs of scientists in the years to come, enabling us to invest our resources wisely and attract users. We have conducted interviews with prospective users about their current and future work to understand and anticipate these needs. Additionally, we are establishing partnerships with prospective users so that we can validate our plans and collaboratively design intuitive interfaces. Because we have been leveraging these partnerships and other UX methods, we have confidence in the requirements we have gathered and are able to build support and generate enthusiasm among collaborators. Our development plans are organized so that we deliver highly in-demand capabilities first, giving our project early wins. Vignette 7: Improving navigability Our scientific software stewardship organization had launched its website but we wanted to establish a more professional look and feel and ensure visitors could easily navigate through our large portfolio of projects. We conducted a heuristic evaluation of the website and took notes on opportunities for improvement. In light of what was found, we made several changes. For example, we added “breadcrumbs” so that visitors will not lose track of where they are and can easily return to other pages higher in the content hierarchy. We also adjusted our navigation in other ways so that our website aligns better with design standards. Finally, we restructured the pages to better organize information. These changes made it easier for users to find what they are looking for because information is where they expect it to be. Vignette 8: Prototyping to support data management At the National Energy Research Scientific Computing Center (NERSC), we have been developing user tools for data management. We provide a Community File System for our users to store data and share it with fellow project members. Efficient use of the file system requires management work, such as keeping track of usage against quota limits and wrangling permissions. In order to make such tasks easier, we developed a set of web-based data management tools. These include a Data Dashboard, which provides visualizations of files in shared directories, the PI Toolbox, which enables PIs to change permissions on all project files, and the Petabyte Data Portal, a file sharing interface. To assess user needs, we began the initial design with a series of user interviews, asking six users from different projects about their biggest needs (e.g., managing permissions) and biggest questions (e.g., who owns this file?). We then performed a task analysis, analyzing the need and frequency for each task for a principal investigator (PI) and for a non-PI project member. This enabled us to prioritize development of the tools that would most likely address frequent needs for many users. For each tool, we used an iterative prototyping and testing approach. After initial design, we created prototypes and used those for task-based usability testing. We tested with paper prototypes as well as interactive prototypes via videoconferencing. After each test, we iterated the design and updated the prototype. Each tool underwent at least three rounds of testing and iteration. After release, we have continued to study our users' needs and add elements to the existing tools to address them. Users frequently comment on how easy the tools are to use, and some have requested our help in replicating them at other centers. Vignette 9: Improving researcher onboarding A team of UX professionals embedded within a precision medicine consortium followed an iterative design approach to improve research onboarding. In addition, the team launched a Researcher Satisfaction survey that generated 938 responses within a year, providing ongoing insights into user needs and pain points. Through targeted UX interventions in researcher onboarding, the team achieved a 26-day reduction in median onboarding time, from 47 days to 21 days between account creation and first workspace use. Importantly, we removed barriers that prevented researchers from accessing critical scientific resources, some of which were not available anywhere else. Strategic design improvements in onboarding included streamlining registration processes from six to four steps, implementing new technologies for identity verification, and redesigning institutional data access request forms based on user feedback. These changes resulted in a 15% decrease in technical support requests and eliminated manual ID verification requests entirely. 139 support requests dropped to zero post-implementation. Vignette 10: Simplifying scientific work ACERetro is a retrosynthesis planning tool developed within the Molecular Maker Lab Institute (MMLI) as part of the AlphaSynthesis platform. While the underlying algorithm introduced a novel search heuristic, early user studies revealed that many researchers found it difficult to understand how input parameters such as reaction type constraints and pathway depth impacted the search space and results. To improve usability, the design team conducted user interviews and task-based evaluations with researchers. Insights from these sessions led to interface updates that clarified terminology, simplified input workflows, and introduced new visualizations to distinguish enzymatic from chemical steps. Explanatory support for SPScore interpretation and route optimization logic was also integrated. These improvements helped bridge the gap between algorithmic innovation and researcher usability, making the tool more approachable for exploring and comparing chemoenzymatic synthesis strategies. Vignette 11: Redesigning a core feature Our team develops a genomics data visualization tool with a core feature that enables users to create a variety of graphs. We knew that this feature was well used but from observational studies we knew that users struggled with the interface. This interface offered only drop down menus to manipulate the X and Y axes, which didn’t relate to the user’s biological questions, such as ‘Do women or men have more frequent mutations in this gene?’. We prototyped three designs based on our observations and watched users complete a task using each design. The results included surprising results, such as users preferring the design with more clicking because they felt empowered to make the graph/chart that they wanted. The design that best satisfied user needs was refined based on feedback, underwent user acceptance testing, and was deployed. Users created more graphs per session with the new interface and they spent more time on the site looking at their graphs. Additionally, the time required to teach how to use this feature was reduced from ~30 minutes to ~10 minutes, and there have been no email questions about the new interface. Vignette 12: Flexibly incorporating UX to shape a product space Our project was developing a software framework to make it easier to see data change—how datasets shift when instruments are reconfigured and software or workflows are updated. Our team employed UX research in multiple rounds to characterize and shape the requirements for a cross environment data change identification tool. We first paired qualitative interviews of domain scientists with data scientists’ own analyses of changes in sample datasets to identify fundamental requirements for our tool. A computer scientist team member then developed an initial version of the tool. We then conducted another round of interviews and leveraged mockups to conduct a simple usability test of the tool’s command line interfaces while also gathering feedback on the potential website interface. This feedback emphasized some core types of information about changes that need to be conveyed, forms of analyses users would want to conduct on demand, and the importance of understandable documentation. Through this work, we were able to iteratively develop a tool well suited for the complex domain analyses in question while also gaining a better theoretical understanding of data change.