BSC scientists contribute to the development of a European HPC workflow platform

16 April 2024

Coordinated by BSC, the eFlows4HPC project enhances complex HPC workflows thanks to big data analytics and AI techniques

HPC Workflows as a Service (HPCWaaS) and eFlows4HPC software stack notably boost scientific simulation performance for natural disasters, climate change and manufacturing processes

A broad spectrum of users from various scientific domains will benefit from this platform, reducing execution times and improving overall productivity

Coordinated by BSC, the eFlows4HPC project has successfully delivered a comprehensive workflow platform and an additional set of services facilitating the seamless integration of High-Performance Computing (HPC) simulation and modelling with big data analytics and machine learning techniques. BSC researchers contributed to this project by defining complex workflows that resulted in enhanced workflow efficiency and increased overall development and deployment productivity. A variety of scientific and industrial domains and other initiatives within the European HPC and scientific applications ecosystem will benefit from this innovative technology, reflecting a wide-reaching and transformative impact of the project.

BSC experts successfully has released an open-source eFlows4HPC software stack that plays a key role in tackling challenges related to natural disasters and climate change, for example. Developed by Rosa M. Badia’s Workflows and Distributed Computing group at BSC, the COMPSs/PyCOMPSs technologies were validated in the different versions of the supercomputer MareNostrum to implement different scientific workflows and applications. Rosa M. Badia, eFlows4HPC coordinator, thinks that “the eFlows4HPC software stack and methodology has been implemented in various editions of the MareNostrum supercomputer, obtaining a notable decrease in time-to-solution for software developers.” Another BSC outcome is the Container Image Creation service (CIC), which automates the generation of HPC-ready containers that simplifies the installation of complex software on HPC systems.

Moreover, BSC researchers from CASE and Earth Sciences Departments have also contributed to the project’s pillars related to natural hazards and climate modelling through their involvement in the CoEs ChEESE, ChEESE-2P and ESiWACE3, respectively, by engaging state-of-the-art seismic simulations and reducing climate modelling time. For natural hazards and as part of ChEESE and ChEESE-2, the developed UCIS4EQ workflow aims to provide insights into the impact of a large-magnitude earthquake event a few minutes after occurs for mitigation and resilience actions. It is a fully automatic HPC workflow orchestrated by PyCOMPSs technology, that has been enhanced to evaluate uncertainty quantification in seismic urgent computing. “The eFlows4HPC software stack has helped us to develop complex workflows for earthquake impact simulation where urgent computing in needed”, says Josep de la Puente, BSC Wave Phenomena Group and eFlows4HPC pillar 3 leader.

Joint collaboration activities have taken place with other BSC projects such as PerMedCoE (personalized medicine), DT-GEO (aiming to build a digital twin components for geophysical extremes) and CAELESTIS project (next generation aircraft design and predictive manufacturing). Finally, the centre also hosted 2 of the 4 community workshops making sure that the developed technology will be adopted by other scientific and industrial communities of the European HPC and scientific applications domain ecosystem. Josep de la Puente continues by saying that “eFlows4HPC methodologies will also be used for the development of workflows in the DT-GEO project and other future projects in geosciences.”

Additional resources:

About eFlows4HPC

After three years of research, eFlows4HPC is a European-funded project with a budget of €7.6M that started on 1 January 2021 and finished in February 2024. Coordinated by BSC (Spain), the project brings together a multidisciplinary consortium: CIMNE (Spain), FZJ (Germany), UPV (Spain), ATOS (France), DtoK Lab (Italy), CMCC (Italy), INRIA (France), SISSA (Italy), PSNC (Poland), UMA (Spain), AWI (Germany), INGV (Italy), ETHZ (Switzerland), Siemens (Germany), and NGI (Norway). The eFlows4HPC project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway. It also received funding from MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957).