IN-CORE - Software Engineering Practices on a Large Scale Project
Panchani, Rashmil; Lee, Jong; Navarro, Christopher; Kooper, Rob; Wang, Chen; Kim, Yong Wook
- Publisher
- Zenodo
- Language
- en
Abstract
The Interdependent Networked Community Resilience Modeling Environment (IN-CORE) is a state-of-the-art computational environment developed by the NIST Center of Excellence for Risk-Based Community Resilience Planning to advance community resilience measurement science and provide a decision support system. IN-CORE integrates various models, including physics-based modeling of infrastructure, network models, data-based socio-economic models, and resilience metrics, to help communities prepare for, mitigate, and rapidly recover from hazard events. IN-CORE has over 28 GitHub repositories, managing various components of the system, including Helm charts used for deployment in a Kubernetes cluster, several Python libraries for modeling, visualization, and community contribution, as well as the Java-based IN-CORE web services. Each software component has continuous build and integration that happens through a series of GitHub actions that help automate the software development process, from Unit testing and Linting to automatic GitHub releases to GitOps for software deployment. While all projects can benefit from this automation, it is particularly beneficial on a large-scale project with many components to help manage the complexity of the entire process, help with quality control, and ensure smooth delivery of releases and deployments. On the modeling side, the IN-CORE development team at NCSA has developed a process for managing the intake of new models from researchers that is dependent on where the researcher is in their development process. For the case where a researcher has code that is ready, they can submit it for review by a research software engineer (RSE). An RSE is assigned to work with the researcher to review their code and finish the implementation in IN-CORE. If a researcher is at an earlier stage of development, they can submit for consultation so that an RSE can help them with library selection, requirements gathering and other implementation details to help get their code ready for inclusion in IN-CORE. Regardless of the readiness, this often involves an iterative process of working with the researcher and communicating with the research team through online meetings, Slack and email. In some cases, there may be scientific questions that need to be resolved and a separate Science Committee (SciCom) is involved to help resolve any issues before the implementation can continue. The software development life cycle is more complex on larger projects with many moving parts. Having good processes in place can help manage this complexity.
Full text
RSE CONTRIBUTION TO IN-CORE Coordinate dependencies across research efforts ●Deliverable 1: Implement analysis and data on IN-CORE ●Deliverable 2: Ensure analyses are interconnected with related analyses Drives specification of input/output of analysis ●Defining data standards and ontology (e.g. building inventory schema) ●Designing the interface for chaining analysis: output becomes input of another analysis Defines analysis input and output specifications ●Designed with both generalization and specialization in mind Bridge academics and practitioners ●Improving models based on community needs ●Enhancing communication of model results to community planners with clearer terminology and refined visualizations. Translate research results into actionable items Enable Community planners to evaluate ●IN-CORE enabling quantitative comparisons of alternative resilience strategies. Rashmil Panchani, Jong Sung Lee, Christopher Navarro, Rob Kooper, Chen Wang, Yong Wook Kim National Center for Supercomputing Applications (NCSA), University of Illinois at Urbana-Champaign, IL, USA IN-CORE - Software Engineering Practices on a Large Scale Project WHAT IS IN-CORE - Advance community resilience science by simulating how hazards (earthquakes, floods, hurricanes and etc.) impact buildings, infrastructure, and people - It incorporates a risk-based approach to decision-making that enables quantitative comparisons of alternative resilience strategies. - On the platform, users can run scientific analyses that model the impact of natural hazards and resiliency against the impact on communities. OVERVIEW Large-scale scientific software projects face challenges in complexity, sustainability, and integration of contributions from diverse researchers. IN-CORE demonstrates how research software engineers (RSEs) can apply modern software engineering practices (CI/CD, containerization, GitOps, structured intake processes) to support scientific innovation at scale. THE SOFTWARE ENGINEERING PRACTICES To manage 28+ repositories, including 3 published python packages, backend services, frontend client code, deployment scripts and etc, IN-CORE uses modern DevOps + RSE practices: Continuous Integration (CI): ●Unit testing ●Code linting ●Automated GitHub releases Continuous Deployment (CD): ●Helm charts & Kubernetes cluster deployments ●GitOps for reproducible rollouts Scientific Code Review Process: ●Consultation stage: RSE works with researchers on requirements, dependencies, libraries. ●Code-ready stage: RSE performs review, testing, and integration. ●SciCom stage (if needed): Scientific committee resolves domain-specific questions. IN-CORE Architecture TAKEWAYS ●Large-scale projects benefit from structured software engineering processes. ●Automation (CI/CD + GitOps) improves quality and reduces integration overhead. ●RSEs bridge the gap between domain researchers and robust, production-ready software. ●A clear intake + communication process ensures sustainable community contributions. IN-CORE Website https://tools.in-core.org/ Support Email [email protected] Sign-Up a Free IN-CORE Account COMMUNICATION CHANNELS Effective collaboration is critical to a successful completion of an analysis. That is why we employ the use of these communication platforms to better understand the needs of a researcher. These are the following communication methods that we currently use: ●Slack → day-to-day questions and async discussions. ●Design meetings → iterative design and integration. ●Clinics → Dedicated sessions where researchers present their analyses, receive feedback, and work with the team on integration challenges. ●SciCom meetings → resolving open research/scientific questions. ●Email → formal documentation & notifications. ●Wiki → Central knowledge base for documentation, guidelines, and shared resources. COMMUNITY ENGAGEMENT VIA SOFTWARE IN-CORE leverages software engineering practices to deliver accessible tools that connect research with community planners and support real-world resilience decisions. Acknowledgement: The research reported herein was supported by the National Institute of Standards and Technology (NIST) (Award No. 70NANB15H044). This support is gratefully acknowledged. The views expressed in this paper are those of the authors and do not necessarily reflect the views of NIST.
