Researchers from the Barcelona Supercomputing Center (BSC) are participating in the European-funded project ADMIRE, which will create an adaptive European storage system – comprising high-performance computing (HPC) simulation, bioinformatics, and artificial intelligence – to optimize data-intensive applications.
ADMIRE will create an active input/output (I/O) software stack that dynamically adjusts computation and storage requirements through intelligent global coordination, malleability of computation and I/O, and the scheduling of storage resources along all levels of the storage hierarchy. The project is coordinated by University Carlos III of Madrid (UC3M) and funded by the European High-Performance Computing Joint Undertaking (EuroHPC JU) and participating states.
As part of the project, BSC researchers in the Storage Systems for Extreme Computing and Distributed Object Management teams will develop a software-defined framework to improve orchestration between key system components and applications.
“For some time, our teams have been researching how to improve data-intensive I/O for HPC and, most importantly, how to integrate it with batch-scheduling decisions for enhanced control of HPC resources. We will contribute to ADMIRE by extending three of our already-ongoing software projects to enable data-intensive HPC I/O: GekkoFS, dataClay and NORNS,” says Alberto Miranda, the BSC principal investigator in the ADMIRE project.
GekkoFS is a user-level file system that allows users to create an on-demand file system for their HPC applications that uses on-node local storage for I/O provisioning. Co-developed with Johannes-Gutenberg University Mainz, the GekkoFS file system has consistently demonstrated outstanding performance results in the IO500 ranking.
dataClay is a distributed data store that enables applications to store and access objects in the same format they have in memory, allowing them to execute object methods within the data store. Finally, NORNS is an infrastructure service that offers asynchronous APIs to applications to simplify the data staging between HPC storage tiers.
“By integrating these three components with ADMIRE’s services for scalable monitoring and control, our goal is to restrict application I/O to node-local storage as much as possible. This so-called ‘I/O containerization’, coupled with a strict definition and enforcement of I/O QoS constraints, will allow us to precisely control when and how an application accesses the cluster’s long-term shared storage system,” adds Alberto Miranda.
“Other components in the ADMIRE framework will provide up-to-date analytics which will allow us to produce precise I/O scheduling solutions that aim to reduce I/O competition between applications. This should improve application performance significantly.”
ADMIRE Overview
Thanks to the integration of these BSC components into the ADMIRE framework, and their extension through the addition of embedded control points, the ADMIRE consortium aims to produce a software-defined storage framework that allows applications from six pillars to better leverage an increasingly complex HPC storage stack. The application pillars are as follows: weather forecasting, molecular dynamics, turbulence simulations, planetary scale cover mapping, brain super-resolution imaging, and software heritage catalogue management and indexing.
About the ADMIRE project
ADMIRE (Adaptive multi-tier intelligent data manager for Exascale) is a European-funded project with a budget of €7.9M that started on 1 April 2021 and will last for three years. Coordinated by UC3M (Spain), the project brings together a multidisciplinary consortium: BSC (Spain), JGU (Germany), TUDA (Germany), MPG (Germany), FZJ (Germany), DDN (France), Paratools (France), INRIA (France), CINI (Italy), CINECA (Italy), E4 (Italy), PSNC (Poland), and KTH (Sweden).
The ADMIRE project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement no. 956748. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, and Sweden.