The Barcelona Supercomputing Center offers to the HPC community a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures.
This release includes the support for a new parameter type, "Stream", which enables the combination of data-flows and task-based workflows in the same application; a new feature that enables the definition of tasks groups with tasks exception management to develop much more dynamic workflows.
Attend a PyCOMPSs tutorial on Monday and live PyCOMPSs demos at the BSC booth #1975.
The Workflows and Distributed Computing team at the Barcelona Supercomputing Center (BSC) is proud to announce a new release, version 2.6 (codename Gardenia), of the programming environment COMPSs.
This version of COMPSs updates the result of the team’s work in the last years on the provision of a set of tools that helps developers to program and execute their applications efficiently on distributed computational infrastructures such as clusters, clouds and container managed platforms. COMPSs is a task-based programming model known for notably improving the performance of large-scale applications by automatically parallelizing their execution.
The new release comes with a new direction type for the tasks' parameters: Commutative, which applied to a group of tasks enables to indicate that the tasks in the group can be executed in any order, and that this change of order does not affect the final result.
The release also adds a new innovative parameter type, "Stream". Stream data, continuous input and output data, can be read or written by tasks, which can also exchange streamed data between them. This new feature enables the combination in the same workflow of the data-flow (tasks exchanging streamed data) and task-based (tasks operating in individual instances of data) paradigms, supporting much more complex Data Science workflows than before.
Another key aspect added in this release is obtained with a set of new features that extend the failure management mechanism added in release 2.5. The new release supports the definition of tasks' groups and the exception management within these groups. Exception can now be thrown from tasks and catched in the main program. As consequence of the exception, the runtime may cancel the remaining tasks in the group and execute the catch code block. These features facilitate the development of much more dynamic type of workflows. Examples of these type of workflows are: convergence detection from the tasks, avoiding global synchronizations; support for speculative tasks; or dynamic exploration of the design space driven by previous tasks results without global synchronizations.
COMPSs 2.6 also support native Python MPI tasks, fine-grain parallelization with OmpSs-2 (C binding) and timeout property for tasks.
COMPSs 2.6 comes with other minor new features, extensions and bug fixes.
The packages and the complete list of features are available in the Downloads page. A virtual appliance is also available to test the functionalities of COMPSs through a step-by-step tutorial that guides the user to develop and execute a set of example applications.
Additionally, a user guide and papers published in relevant conferences and journals are available.
COMPSs at SC19
At SC19, PyCOMPSs/COMPSs will be part of the tutorial Practical Persistent Memory Programming, on Monday November 18th at 8.30AM, and will have a presence at the BSC booth, with presentations and demos. Live demos on the use of PyCOMPSs with Jupyter notebooks will be delivered at the BSC booth #1975 with the following program:
- Tuesday Nov 19th, Demo PyCOMPSs (14.00 - 14.30) - Accelerating parallel code with PyCOMPSs and Numba
- Wednesday Nov 20th, Demo PyCOMPSs (13:00 - 13:30) - Parallel machine learning with dislib
- Thursday Nov 21th, Demo PyCOMPSs (11.00 - 11.30) - Fault-tolerance mechanisms for dynamic PyCOMPSs workflows
About COMPSs
COMPSs has been available for the last years for the MareNostrum supercomputer and Spanish Supercomputing Network users, and it has been adopted in several research projects such as OPTIMIS, VENUS-C, EUBrazil OpenBio, EUBrazil CloudConnect, EUBra-BIGSEA, transplant, MUG, EGI, ASCETIC and TANGO. In these projects, COMPSs has been applied to implement use cases provided by different communities across diverse disciplines as biomedicine, engineering, biodiversity, chemistry, astrophysics and earth sciences. Currently it is also under extension and adoption in applications in the projects NEXTGenIO, mF2C, CLASS, ELASTIC, ExaQUte, LANDSUPPORT, the BioExcel CoE, and the EXPERTISE ETN, as well as in a research contract with FUJITSU.
COMPSs has had around 1000 downloads last year and is used by around 20 groups in real applications. COMPSs has recently attracted interest from areas such as image recognition, genomics and biodiversity, where specific courses and dissemination actions have been performed.
For more information on COMPSs please visit our webpage: http://www.bsc.es/compss
About the team
The Workflow and Distributed Computing team at the Barcelona Supercomputing Center aims to offer tools and mechanisms that enable the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources in a transparent way. The research done in this team is based in the former expertise of the group, and extending it towards the aspects of distributed computing that can benefit from this expertise. The team at BSC has a strong focus on programming models and resource management and scheduling in distributed computing environments.