Preliminary Results from the AI Blueprint Factory

Preliminary Results from the AI Blueprint Factory Jeanette Sperhac San Diego Supercomputer Center University of California San Diego La Jolla, CA, USA ORCID: 0000-0003-0436-9318 Sandra Gesing San Diego Supercomputer Center University of California San Diego La Jolla, CA, USA ORCID: 0000-0002-6051-0673 Robert Quick Pervasive Technology Institute Indiana University Bloomington, IN, USA ORCID: 0000-0002-0994-728X Joe Stubbs Texas Advanced Computing Center University of Texas at Austin Austin, TX, USA ORCID: 0000-0002-8644-0300 Christian Garcia Texas Advanced Computing Center University of Texas at Austin Austin, TX, USA [email protected]xas.edu Chi-kwan Chan Steward Observatory Department of Astronomy University of Arizona Tucson, AZ, USA ORCID: 0000-0001-6337-6126 Ewa Deelman Information Sciences Institute University of Southern California Marina Del Rey, CA, USA ORCID: 0000-0001-5106-503X Amy Connolly Department of Physics The Ohio State University Columbus, OH, USA ORCID: 0000-0003-0049-5448 Huanmei Wu College of Public Health Temple University Philadelphia, PA, USA ORCID: 0000-0003-0346-6044 Abstract—Advancements in Artificial Intelligence (AI), including generative systems like ChatGPT and DeepSeek, are motivating a revolution in academic research computing. Academic researchers are turning to AI to make fundamental strides in their research, creating and providing new models and algorithms, AI-ready datasets, AI-relevant cyberinfrastructure, and more. However, the academic sphere lags behind the private sector in AI resource availability and adoption. Science gateways have long facilitated academic research by simplifying the access and use of advanced computing resources, and promoting collaboration. Can science gateways help academic research communities by promoting access to advanced AI resources, new algorithms and models, and AI-ready datasets? The NSF Center of Excellence for Science Gateways (SGX3) founded an initiative called the AI Blueprint Factory to address this question. The Blueprint Factory will identify technical capabilities required to support evolving scientific and computing needs, and consider how science gateways can help. The purpose of the AI Blueprint Factory is to ascertain the AI needs of research communities that use national-scale computing infrastructure, and to make 5 to 10 year forecasts of needed features and resources. Its study team is interviewing researchers across disciplines and seniority levels to determine perceived needs, opportunities, and gaps in AI research support. The focus is not on supporting core AI research, but rather on fostering the utilization of AI techniques in domain science research. In this paper, we introduce the study and report preliminary findings from the first set of researcher interviews. Index Terms—Science Gateways, cyberinfrastructure, high performance computing, machine learning, artificial intelligence, scientific software. We acknowledge the support of National Science Foundation (NSF) OAC 2231406 (SGX3) and NSF 2127548 (CI Compass). I. INTRODUCTION Artificial Intelligence (AI) has been influencing science and industry since the 1950s. [1] By 2011, the speed of GPUs increased to the point of exercising a significant advantage over CPUs for running deep learning algorithms. [2] Recent excitement around AI arises from ChatGPT [3], DeepSeek [4], and other large language models (LLMs) which have gained attention due to public user interfaces, ease of use, and ready availability. AI methods are promising for finding novel solutions in research computing and reshaping many aspects of the field. Meanwhile, over two decades, science gateways have become part of the research computing landscape. [5] Gateways help to address the need for usability of computational solutions, sharing simulations and data, and reproducible research. The NSF Center of Excellence for Science Gateways (SGX3) [6] is studying how the community pursuing AI-driven research can benefit from science gateways, and in turn, the science gateways community can benefit from AI techniques in research computing. This study and other SGX3 Blueprint Factories are intensive endeavors spanning approximately 18 months, focused on ascertaining the technical capabilities necessary for science gateways to support the future requirements of specific scientific domains or computing resources. [7] This paper introduces the AI Blueprint Factory and presents preliminary results from its first round of researcher interviews. Section II provides background about science gateways and SGX3. Section III introduces the study. Section IV describes data collection and initial analysis. Section V presents the preliminary results. Section VI outlines future work. II. BACKGROUND Science gateways are platforms that facilitate access to data, software, computing services, collaborative features, and specialized equipment tailored to scientific and engineering disciplines. They simplify the complexities of using research computing infrastructure, allowing scientists and educators to focus on research and teaching. Science gateways have seen significant adoption and influence in domains including neurology, phylogenetics, nanotechnology, and natural disasters. A. SGX3 The Science Gateways Community Institute (SGCI) and its successor, SGX3, have supported science gateway owners, users, and developers since 2016. [8] The SGX3 project has been funded since August 2022 to help advance academic science gateways by providing services including the Blueprint Factories. This paper describes one such study. B. Science Gateways and AI AI-enabled research is triggering a proliferation of publicly funded science gateways such as Foundry ML, Garden, Diamond, Illinois Chat, and the NAIRR Portal, which aim to offer researchers and educators access to AI-ready models, datasets, managed LLMs, and sometimes compute resources. [9]–[13] Even commercial AI platforms such as HuggingFace, Modal, and Cargo are arguably science gateways, delivering access to models, compute cycles, and data, sometimes at a cost. [14]– [16] Furthering collaborations between science gateway and AI communities will continue to drive innovation, addressing the usability and accessibility of AI resources and features for research. How can science gateways better encompass and integrate the burgeoning space of AI platforms, private and public? What features and services will the AI-enabled research community need from science gateways into the future? SGX3 seeks to address such issues via the AI Blueprint Factory. C. Blueprint Factories The Blueprint Factory concept is part of an SGX3 objective to envision the future of research computing. A Blueprint Factory study collects data from focus groups and researcher interviews to identify emerging needs and develop a forwardlooking blueprint of technical capabilities for science gateways. Each such study spans approximately 18 months and is guided by a core team consisting of cyberinfrastructure (CI) professionals and science domain experts. SGX3 aims to conduct a total of seven Blueprint Factories to explore topics that include CI sustainability and quantum computing. III. THE AI BLUEPRINT FACTORY Though AI is increasingly prevalent in contemporary research, its use across the scientific disciplines is still in its early stages. The full implications of AI on research, science gateways, and computing resources remain unclear. The AI Blueprint Factory is intended to help chart the path forward. A. Motivation A study such as the AI Blueprint Factory is relevant for research computing because the needs of AI workflows differ from those of traditional high-performance computing (HPC) computations that are so common in research computing. For example, AI model training can be laborious, requiring considerable time on specialized processors. Model training tasks are iterative and interactive, with natural checkpoints, and require different model validation processes than typical HPC workloads. Moreover, data management is a pivotal concern for AI workloads. Training AI models often requires extensive datasets whose size makes transferring them to needed compute resources a challenge. In addition, AI models are frequently used for data recall and are therefore located close to the data source. AI-based models may be updated frequently with incremental training, necessitating careful management of data provenance, which entails tracking and characterizing both the data used to train specific model versions and the model versions used. Ethical considerations accompany the proliferation of AI in scientific research. The handling and analysis of vast amounts of data raise concerns about data security, privacy, and the potential for bias in AI models. Researchers and institutions must navigate these challenges responsibly, ensuring that data is used in ways that respect individual privacy, and that AI models are trained and deployed within the bounds in which they were intended, without reinforcing existing biases. This suggests that data privacy, model suitability and the orchestration of training, validation, data management, and recall in AI research can be complex. These challenges may be eased through the use of science gateways. The AI Blueprint Factory explores how researchers across disciplines are using AI, what challenges they face, and how science gateways might help address those challenges. The focus is not on advancing core AI research itself, but on enabling domain scientists to more effectively apply AI methods in their work. Furthermore, we consider how science gateways can broaden access to AI techniques to reach more domain-specific researchers. B. Organization A core team consisting of 3 domain experts and 5 CI experts was convened to guide the AI Blueprint Factory. These core team members either use AI in scientific research, or build and support CI for scientific research. At the beginning of the study, the team held a focus group about AI use in scientific research. In the resulting Birds of a Feather (BoF) session, held at Practice and Experience in Advanced Research Computing (PEARC) 2023, members of the research computing community discussed prominent issues around AI in research. Conclusions derived from the analysis of the BoF session were presented at PEARC 2024. [17] These conclusions helped establish a baseline understanding of researcher concerns, challenges, and goals with respect to AI use in their work. TABLE I DEMOGRAPHICS OF INITIAL AI BLUEPRINT FACTORY INTERVIEWEES US Region Count Midwest 4 West 3 South 1 Seniority Count Graduate Student 3 Early Career 3 Mid Career 2 Institutional Classification Count R1 6 R2 1 National Lab 1 Research Discipline Count Computer Science 3 Materials Science 3 Social Science 1 Medicine 1 Next, the AI Blueprint Factory core team compiled a list of potential interview candidates from a variety of scientific disciplines, seniority levels, and home institutions. All interview candidates conduct research that uses AI techniques and methods. The team drafted an interview guide encouraging free-form discussion and based around core ideas surfaced in the Science Gateways AI BoF session described above. A recruitment letter and a study information sheet were prepared to help enlist potential participants, and team members secured Institutional Review Board (IRB) Exempt approval from their home institutions. IV. DATA COLLECTION AND INITIAL ANALYSIS For the data collection phase of the project, the AI Blueprint Factory team will conduct at least 30 individual interviews in total. Questions asked of each researcher cover the subtypes of AI used, the research domain, computing and data sources utilized, data and workflow management, interfaces used, and concerns about AI ethics and use. Interviews are performed remotely, using Zoom software with its AI Companion feature enabled to record and automatically transcribe each session. [18] Following an initial set of interviews, the team reviewed and adapted the interview guide to reflect lessons learned. In this paper, we report only on preliminary conclusions drawn from an initial set of 8 interviews. Data analysis consists of deidentifying, summarizing, and synthesizing the themes uncovered in the interviews. Deidentifying the transcripts is performed offline with scripts written in Python. Deidentified data are summarized and given additional context, and provided as input to managed LLMs which can be repeatedly queried to help summarize and characterize the contents. [12], [19], [20] The researchers interviewed in the preliminary set of interviews represented a diversity of seniorities, US regions, and research disciplines. Table I shows that the seniority levels of the interviewees were relatively evenly distributed, with 3 grad students, 3 early career, and 2 mid-career researchers represented. 5 out of 8 interviewees (62.5%) were affiliated with R1 institutions, with the majority (62.5%) located at institutions in the Midwestern and Western regions of the US. There is some variation in the research disciplines, with 75% from Materials Science and Computer Science. The racial distribution of the preliminary interviewees was 37% Asian, 25% Black, and 37% White. By sex, 25% of interviewees were female and 75% were male. This paper describes only the first set of interviews; a total of at least 30 will be completed to provide sufficient data for the final report. V. PRELIMINARY RESULTS In these initial interviews, researchers were asked about their work with AI. Researchers described their projects and the computing resources, tools, and software they use. They noted opportunities, challenges, and gaps they observe in the national research infrastructure and the software and cyberinfrastructure offerings they use. Despite the small sample, researcher responses echoed themes from the Science Gateways AI BoF analysis presented at PEARC ’24. (See Section III-B). [17] The AI techniques and applications used by this first group of interviewees vary considerably. All researchers interviewed use more than one AI technique in their work, including: •Machine Learning (ML): 75% of the researchers mention using techniques such as supervised and unsupervised learning. •Natural Language Processing (NLP): Some researchers (37.5%) mention using techniques such as text analysis to analyze natural language data. •Computer Vision: 25% of the researchers mention using techniques such as image recognition and object detection on visual data. •Deep learning: 25% of the researchers mention using deep learning techniques, including generative AI. Compute resources and tools: The researchers in this preliminary set reported using a variety of computing resources; some researchers used several types of resources. Private cloud resources, such as those provided by MS Azure and Amazon Web Services, university and lab resources, and national resources, such as those supported by NSF ACCESS and DoE, are utilized by these interviewees in equal numbers. While some researchers prefer to use resources such as ACCESS’ Jetstream2 for AI work, requiring them to exercise considerable knowledge in set up, others are happy to see the ascendance of turn-key cloud solutions, enabling newcomers to AI research to participate without first becoming an expert in system tooling. More than half of the participants normally use the command line to interact with computing resources (62%), though many also use notebooks. Even within this small group, researchers reported using a wide array of tools, frameworks, and platforms. Several use frameworks such as PyTorch and TensorFlow, and a combination of academic and commercial platforms and tools, including HuggingFace, Modal, Globus, and Foundry ML. Researchers commonly develop their own libraries and work- flow tools to reflect the specific needs of their discipline and research problem. Data: Interviewees consistently noted that data management requires substantial time and effort for their AI workloads. They mentioned the importance of data and model provenance, such as tracking data origins, transformations, and versions; data labeling and annotation; data storage and retrieval for large datasets; and data privacy and security. Several researchers interviewed are subject to stringent restrictions on their data, such as HIPAA. They indicated that these data controls, and the attention to data privacy and safety, are appropriate and sufficient. Opportunities: Interviewees highlighted numerous opportunities for improving research and AI. These overlap somewhat with the data concerns they raised. Frequent mentions include: •Data management: Data storage, labeling, provenance, and archival are significant challenges in AI research. Software and workflow tools that support FAIR (Findable, Accessible, Interoperable, and Reusable) data practice could streamline this process. •Discoverability and accessibility of computing resources: A streamlined allocations process, and ideally on-demand access, is needed for public computing resources. Researchers also recommended improving discoverability of computing resources and documentation about their features. •Reproducibility and reusability of AI models: Researchers discussed challenges in making models testable, transparent, explainable, and reusable. While some of these challenges are due to the black box nature of deep learning models, better metadata, runtimes, and cyberinfrastructure can help support these goals. •Targeted use cases and examples: Researchers emphasized the importance of making models and datasets accessible through sharing platforms. Concrete use cases and examples were seen as essential to demonstrate the capabilities of such platforms/gateways. Though they pursue different research problems and use different compute resources, the researchers in this first round of AI Blueprint Factory interviews share similar concerns about the challenges of using AI in research. They show a common interest in improved availability of compute resources, straightforward handling and integration of data, reusability of AI models, and continued attention to data privacy and security. These early findings were substantially in line with the Science Gateways AI BoF analysis presented at PEARC’24. (See Section III-B) [17] VI. FUTURE WORK This preliminary report summarizes results from the initial set of researcher interviews performed for SGX3’s AI Blueprint Factory. The study will incorporate approximately 20 additional interviews with domain researchers representing a variety of disciplines, institution types, and US geographic regions. Following the completion of all interviews, a detailed analysis will describe how researchers use AI in their work, and identify common opportunities, concerns, and pain points. The overarching purpose of the AI Blueprint Factory is to understand the AI needs of research communities that use national-scale computing infrastructure. With a complete set of interviews analyzed, the study will forecast features and resources needed to support AI-enabled research over the next 5 to 10 years. These conclusions will then be published in a whitepaper. SGX3 intends this work to inform future access to hardware and software, training, functionality, and features to support AI-driven academic research. ACKNOWLEDGMENT We thank the researchers who have generously volunteered their time and insights in AI Blueprint Factory interviews. REFERENCES [1] A. M. Turing, “Computing machinery and intelligence,” Mind, vol. 49, no. 236, pp. 433–460, 1950. [2] K. D. Foote, “A Brief History of Deep Learning,” 2022. [3] OpenAI, “Introducing ChatGPT,” 2022. [Online]. Available: https://openai.com/blog/chatgpt [4] DeepSeek-AI, D. Guo, D. Yang, and H. Z. et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948 [5] S. Gesing, C. Stirm, M. Zentner, M. Dahan, and L. Hayden, “SGX3: Novel concepts to enhance knowledge and extend the community around science gateways,” in ”Science Gateways 2023 Annual Conference”. Geneva, Switzerland: Zenodo, 2023, pp. 1–5. [Online]. Available: https://zenodo.org/records/10034892 [6] SGX3, “Science Gateways Institute and SGX3 (SGCI/SGX3),” 2022. [Online]. Available: https://sciencegateways.org [7] ——. (2023) Blueprint Factories. SGX3. [Online]. Available: https://sciencegateways.org/our-services/blueprint-factories [8] N. Wilkins-Diehr, M. Zentner, M. Pierce, M. Dahan, K. Lawrence, L. Hayden, and N. Mullinix, “The science gateways community institute at two years,” in Proceedings of the Practice and Experience on Advanced Research Computing, ser. PEARC ’18, XSEDE. New York, NY, USA: Association for Computing Machinery, 2018. [Online]. Available: https://doi.org/10.1145/3219104.3219142 [9] Foundry ML, “Foundry ML,” https://foundry-ml.org/, 2025, accessed: 2025-03-19. [10] thegardens.ai, “Garden,” https://thegardens.ai, 2025, accessed: 2025-0319. [11] Diamond.io, “Diamond,” https://diamond.io, 2025, accessed: 2025-0319. [12] University of Illinois Urbana-Champaign, “Illinois Chat (UIUC.Chat),” online, Urbana-Champaign, IL, USA, 2025. [Online]. Available: https://uiuc.chat [13] San Diego Supercomputer Center, “NAIRR Pilot Portal,” online, San Diego, CA, USA, 2025. [Online]. Available: http://nairrportal.org [14] HuggingFace, “HuggingFace,” https://huggingface.co, 2025, accessed: 2025-03-19. [15] Modal, “Modal,” https://modal.com, 2025, accessed: 2025-03-19. [16] Cargo, “Cargo,” https://www.getcargo.io, 2025, accessed: 2025-03-19. [17] J. Sperhac, S. Gesing, M. Zentner, C. Stirm, R. Quick, and J. Stubbs, “Ai and science gateways: A promising combination for accelerating science and research computing,” in PEARC ’24: Practice and Experience in Advanced Research Computing, 2024. [18] Zoom Video Communications, “Zoom AI Companion,” Version 6.3.6 (6315), San Jose, CA, USA, 2025. [Online]. Available: http://zoom.com [19] San Diego Supercomputer Center, “SDSC LLM,” Version 202502-07, San Diego, CA, USA, 2025. [Online]. Available: https://doi.org/10.71469/B14W2B [20] National Research Platform (NRP), “NRP-Managed LLMs,” Version 2025-05-14, San Diego, CA, USA, 2025. [Online]. Available: https://internationalresearchplatform.org/documentation/userdocs/ai/llmmanaged/