It has become more than evident by now that the increasing complexity and resource intensiveness of performing state-of-the-art Earth System Science (ESS), be it from a modeling or a pure data collection and analysis perspective, requires tools and methods to orchestrate, record and reproduce the technical and scientific process. To this end, workflows are the fundamental tool for scaling, recording, and reproducing both Earth System Model (ESM) simulations and large-volume data handling and analyses.
With the increase in the complexity of computational systems and data handling tasks, such as heterogeneous compute environments, federated access requirements, and sometimes even restrictive policies for data movement, there is a necessity to develop advanced orchestration capabilities to automate the execution of workflows. Moreover, the community is confronted with the challenge of enabling the reproducibility of these workflows to ensure the reproducibility of the scientific output in a FAIR (Findable, Accessible, Interoperable, and Reusable) manner. The aim is to improve data management practices in a data-intensive world.
This session will explore the latest advances in workflow management systems, concepts, and techniques linked to high-performance computing (HPC), data processing and analytics, the use of federated infrastructures and artificial intelligence (AI) application handling in ESS. We will discuss how workflows can manage otherwise unmanageable data volumes and complexities based on concrete use cases of major European and international initiatives pushing the boundaries of what is technically possible and contributing to research and development of workflow methods (such as Destination Earth, DT-GEO, EDITO and others).
On these topics, we invite contributions from researchers as well as data and computational experts presenting current scientific workflow approaches developed, offered and applied to enable and perform cutting edge research in ESS.
Richard Hofmeister