BSC releases COMPSs version 2.0 at SC16

11 November 2016

This new major release includes new features such as the integration with new persistent storage solutions, integration with the OmpSs programming model including support for heterogeneous devices (GPUs) and a new scheduling infrastructure.

The Workflows and Distributed Computing team at the Barcelona Supercomputing Center is proud to announce a new release, version 2.0, of the programming environment COMPSs.

This version of COMPSs, available from today, updates the result of the team’s work in the last years on the provision of a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures such as clusters, grids and clouds. COMPSs is a task based programming model known for notably improving the performance of large scale applications by automatically parallelizing their execution.

COMPSs has been available for the last years to the MareNostrum supercomputer users and to the Spanish Supercomputing Network and has been adopted in several research projects such as OPTIMIS, VENUS-C, EUBrazil OpenBio, EUBrazil CloudConnect, transPLANT and EGI. In these projects COMPSs has been applied to implement use cases provided by different communities across diverse disciplines as biomedicine, engineering, biodiversity, chemistry, astrophysics and earth sciences. Currently it’s also under extension and adoption in applications in the projects ASCETIC, EUBRA BIGSEA, EUROSERVER, NEXTGenIO, MUG, TANGO, the CoE BioExcel, the BSC Severo Ochoa program and the Human Brain Project flagship. COMPSs will also be further developed and used in the recently accepted H2020 funded projects: mF2C and the EXPERTISE ITN.

The new release comes with major features such as the integration with new persistent storage solutions (i.e. dataClay, or Hecuba), integration with the OmpSs programming model including support for heterogeneous devices (GPUs) and a new scheduling infrastructure. 

The integration with storage solutions which enables the storage of memory objects in persistent memory in a transparent way for COMPSs applications. This will enable in the future the storage of memory objects in persistent storage devices, such as NVRAM or Storage Class Memories (SCM). This new feature will support the development of dynamic workflows of HPC simulations and data analytics in an integrated way.

dataClay is a storage platforms that enables programs to store data and code in a single abstraction in the form of objects and methods (as in OO programming models). Furthermore, structured data can be accessed using different views both with respect to the data and the model of the data depending on the application accessing the data. Currently dataClay is in a pre-release state and supports Java and Python applications.

Hecuba is a set of tools and interfaces that implements a simple and efficient access to data stores for big data applications and that helps developers to improve their productivity. Using Hecuba, the applications can access the data like regular objects stored in memory and Hecuba translates this code at runtime to access the underlying storage system. In addition, Hecuba implements optimizations to favour data locality and to reduce the number of interactions with the backing storage and thus, to speed up the accesses to data. The current implementation of Hecuba supports Python applications that store data in a key-value database (Apache Cassandra).

With regard the computing infrastructures, the new release comes with support for elasticity with Docker and Mesos, and support for SLURM clusters. COMPSs 2.0 comes with other minor new features, extensions and bug fixes.

COMPSs has had around 1000 downloads last year and is used by around 20 groups in real applications. COMPSs has recently attracted interest from areas such as image recognition, genomics and biodiversity, where specific courses and dissemination actions have been performed.

During last years, the team efforts have been focusing on the nowadays-emerging virtualization technologies, adopted by cloud environments. In such systems, COMPSs provides scalability and elasticity features by dynamically adapting the number of resources to the actual workload.

COMPSs is interoperable with both public and private cloud providers like Amazon EC2, Google Cloud, OpenNebula and with OCCI compliant offerings.

The packages and the complete list of features are available in the Downloads page. A virtual appliance is also available to test the functionalities of COMPSs through a step-by-step tutorial that guides the user to develop and execute a set of example applications.

Additionally, a user guide and papers published in relevant conferences and journals are available.

For more information on COMPSs please visit our webpage: www.bsc.es/compss