Mission Science Data Systems (A Mission-Specific Template)

Mission Science Data Systems (A Mission-Specific Template) Mission Science Data Systems (A Mission-Specific Template) Intended Usage Abstract 1 Introduction 1.1 SDS Overview 1.1.1 Background 1.1.2 Current State/Status of SDS 1.1.3 Community and Audience 1.2 Mission Objectives 1.2.1 Summary 1.2.2 Mission References 1.3 Instrumentation 1.3.1 Mission Observables 1.3.2 Quantities Measured 1.3.3 Measurement Techniques 1.3.4 Instrumentation References 1.4 SDS in Mission Context 2 Mission Needs 2.1 Introduction / Summary 2.2 Programmatic Constraints 2.2.1 Ground Systems 2.2.2 Data Volumes and Scalability 2.3 Archive requirements 2.4 Open Science Requirements 3 Science Data System Design 3.1 Architecture Overview 3.2 Data Products 3.2.1 Scientific Deliverables 3.2.2 Data product design philosophy 3.2.3 Archive 3.3 Computing Infrastructure 3.3.1 Hosting Strategy 3.3.2 Information Technology 3.3.3 Operating Authority 3.3.4 Disaster Recovery and Continuity of Operations 3.4 Components 3.5 Data Flow 3.5.1 Telemetry 3.5.2 Ingest 3.5.3 Processing Pipelines and Controls 3.5.4 Science Processing Codes 3.5.5 Data Management 3.5.6 Archive 3.6 Instrument Monitoring and Trending 3.7 Development 3.7.1 Development Team 3.7.2 Team Management 3.7.3 Project Management 3.7.4 Testing Approach 4 Documentation and Support 4.1 Software 4.1.1 Software Documentation 4.1.2 SDS Operator Documentation 4.2 Mission Science and User Documentation 4.3 SDS Support 5 Mission Data and Software Usage Policies OR Open Science 5.1 Rules and Guidelines 5.2 Mission Practices 6 Sustainment and Modernization 6.1 Development and Maintenance Progression 6.2 Operations 6.3 Migrations, updates, and technical debt 7 Lessons Learned 7.1 Example: Used Great Tool for Decom 8 Conclusion Acronyms / Glossary References Intended Usage This document template is a community effort to publish Science Data Systems (SDS) existing in industry and the decisions, technologies, and people that pioneer this landscape. We encourage you and your team, regardless of which Science Mission Directorate (SMD) mission area you represent, to use this as a standardized guide to “tell your story” so that the community may benefit and future missions might learn and utilize these technological advances rather than introducing further redundancy. In 2024 and 2025, three community workshops brought together participants from a range of institutions, mission areas, and system/data scales used in scientific research. These efforts reflect a shared commitment to developing common frameworks, interfaces, and architecture from the diverse systems already used. An introductory report within this special edition will highlight these workshops and summarize key findings. The intended audience of these submissions are fellow designers and developers of SDS. Note: From this point forward, the term SDS will be used instead of other analogous entities (e.g., Science Operations Center (SOC)). Abstract Provide the desire for including your mission’s SDS in this collection and the needs of mission systems to be considered collectively rather than individually. Brief summary of the paper / high-level points should be included (no cliffhangers)--someone reading abstract / intro / conclusion should have a good idea of the paper. 1 Introduction 1.1 SDS Overview 1.1.1 Background - Relevant heritage (and missions) - What the audience needs to know to understand the SDS 1.1.2 Current State/Status of SDS - Mission phase and state of the system (is it under development, currently operating, undergoing modernization efforts, etc.) 1.1.3 Community and Audience - What is the community that uses/benefits from the SDS? - Who would especially benefit from reading this paper? 1.2 Mission Objectives 1.2.1 Summary - Human-readable plain language summary of the mission, high level scientific and non-scientific objectives 1.2.2 Mission References - Table is optional - Cite mission paper - Cite other relevant publicly-available items (e.g., OSDMP, DMP) - Other examples: “first light” paper or mission-defining science results; repository for the SDS if developed in the open (even better, concept DOI); etc. 1.3 Instrumentation 1.3.1 Mission Observables 1.3.2 Quantities Measured - Table of quantities measured - High Level - Reference Section 2 detailed descriptions 1.3.3 Measurement Techniques 1.3.4 Instrumentation References - Cite instrument papers 1.4 SDS in Mission Context Potential topics to cover: (non-exhaustive) - PI-lead or otherwise? - SDS unified for the mission or multiple (instrument, suite, etc.)? - Division of labor between SDS, instrument teams, mission facilities for writing code, validation of products, etc. - Relationship with MOC - Funding source(s)? - Size/class of mission - Where the SDS "sits" in the ground segment (maybe a high level ground systems data flow diagram) - Heritage / history of the relevant research / instrument team - How early were RSE's involved in the mission planning / conception process? 2 Mission Needs 2.1 Introduction / Summary High level summary of requirements/desirements (what you got going into it) Can also include a table of key driving requirements 2.2 Programmatic Constraints Restrictions on launch date, cost caps, decision to have system complete before launch, etc. Export controlled information and need to partition any other sensitive / proprietary information 2.2.1 Ground Systems - Operations - Payload Operation Center (POC) - Science Operation Center (SOC) - Data Retrieval - Scientific Planning, Commanding, and Optimization - Multiple streams of data, e.g. operational SWx stream 2.2.2 Data Volumes and Scalability - Data Volumes - Rates and Latencies - Latency between acquisition and downlink, and between downlink and delivery… - Burst / survey, SITL, etc - Cost Considerations - Scalability 2.3 Archive requirements What archive to deliver to, what requirements do they levy on file formats and such? 2.4 Open Science Requirements Reference any requirements for open science (e.g. SPD-41a or similar policies) that apply to your mission or SDS. Institutional restrictions, encouragement, and/or processes (tech transfer) for release of data and software; additional funding or projects supporting open science. 3 Science Data System Design Include discussion of how design addresses mission needs from section 2 Include relevant trade studies. Discuss as-built design. Summarize relevant changes from initial design to as-built, to be referenced in lessons learned. 3.1 Architecture Overview Architecture diagram(s), internal and external interfaces 3.2 Data Products 3.2.1 Scientific Deliverables - Collections - Data Level definitions - Level, definition from the DAAC, the way you interpreted the definition, and any ways you were not able to make that definition work for your scenario 3.2.2 Data product design philosophy ● Easy to reuse for derived products, v independently usable w/o additional work? ● How many products? ● What goes in what products? ● Feed this into lessons learned, how did these decisions work out? 3.2.3 Archive - Which products did you decide to archive? - Were there any specific data needs for archival? (any specific metadata needed, any label files needed, etc.) - How we generate the data/metadata files and interact with the archive (requirements are covered above) - Link to the data holdings 3.3 Computing Infrastructure 3.3.1 Hosting Strategy - Cloud, on-prem, or hybrid 3.3.2 Information Technology - IT Support - Security (DMZ, public access) 3.3.3 Operating Authority ATOs or similar required? 3.3.4 Disaster Recovery and Continuity of Operations 3.4 Components Note whether the components are open-source, proprietary, reused, etc. 3.5 Data Flow Note whether the tools are open-source, proprietary, reused, etc. Note any difference in data streams, e.g. a near-time space weather product v. standard 3.5.1 Telemetry - Volume - Dimensionality - Rate(s) and frequency of downlink - Format (CCSDS format or not, compression, etc.) 3.5.2 Ingest - Data Transfers - Downlink: how data gets to the mission team/operations 3.5.3 Processing Pipelines and Controls - Automation - Triggers - Dependency management - Reprocessing - Semantic Versioning - Structure (including ancillary files) - How data gets to the end user - Interfaces/Gateways - Open Science (SPD-41A) - FAIR principles - Out-of-loop processing - Anomaly Investigation - Refinement and debugging 3.5.4 Science Processing Codes Brief summary and any points of interest for the data production codes (e.g. L1 -> L2) 3.5.5 Data Management - Versioning - Storage - Inventory/Catalog/Databases - Metadata References - Citations to all software (GitHUb/Zenodo DOIs with specific versions) - OSDMPs/DMPs - Related instrument/mission papers