Skip to main content

File Systems

caution

It is your responsibility as a user of our facilities to backup all your critical data. We only guarantee a daily backup of user data under /gpfs/home. Any other backup should only be done exceptionally under demand of the interested user.

Each user has several areas of disk space for storing files. These areas may have size or time limits, please read carefully all this section to know about the policy of usage of each of these filesystems. There are 3 different types of storage available inside a node:

  • GPFS filesystems: GPFS is a distributed networked filesystem which can be accessed from all the nodes. On this machine, it is mounted on "/slgpfs".
  • Local hard drive: Every node has an internal hard drive
  • Root filesystem: Is the filesystem where the operating system resides

GPFS Filesystem​

The IBM General Parallel File System (GPFS) is a high-performance shared-disk file system providing fast, reliable data access from all nodes of the cluster to a global filesystem. GPFS allows parallel applications simultaneous access to a set of files (even a single file) from any node that has the GPFS file system mounted while providing a high level of control over all file system operations. In addition, GPFS can read or write large blocks of data in a single I/O operation, thereby minimizing overhead.

An incremental backup will be performed daily only for /slgpfs/home.

These are the GPFS filesystems available in the machine from all nodes:

  • /apps: Over this filesystem will reside the applications and libraries that have already been installed on the machine. Take a look at the directories to know the applications available for general use.

  • /slgpfs/home: This filesystem has the home directories of all the users, and when you log in you start in your home directory by default. Every user will have their own home directory to store own developed sources and their personal data. A default quota will be enforced on all users to limit the amount of data stored there. Also, it is highly discouraged to run jobs from this filesystem. Please run your jobs on your group's /slgpfs/projects or /slgpfs/scratch instead.

  • /slgpfs/projects: In addition to the home directory, there is a directory in /slgpfs/projects for each group of users. For instance, the group bsc01 will have a /slgpfs/projects/bsc01 directory ready to use. This space is intended to store data that needs to be shared between the users of the same group or project. A quota per group will be enforced depending on the space assigned by Access Committee. It is the project's manager responsibility to determine and coordinate the better use of this space, and how it is distributed or shared between their users.

  • /slgpfs/scratch: Each user will have a directory over /slgpfs/scratch. Its intended use is to store temporary files of your jobs during their execution. A quota per group will be enforced depending on the space assigned.

GPFS Tapes​

Tapes is a mid-long term storage filesystem that provides 400 PB of total space. You can access tapes from the Data Transfer Machine under /gpfs/tapes/hpc/your_group.

info

There is no backup of this filesystem. The user is responsible for adequately managing the data stored in it.

Hierarchical Storage Management (HSM) is a data storage technique that automatically moves data between high-cost and low-cost storage media. At BSC, the filesystem using HSM is the one mounted at /gpfs/tapes/hpc (not mounted on Starlife nodes), and the two types of storage are GPFS (high-cost, low latency) and Tapes (low-cost, high latency).

HSM System Overview​

Hardware

  • 2 x IBM TS4500
  • 21000 20TB Enterprise Tapes
  • 64 Drives
  • 8 Spectrum Archive Servers

Software

  • IBM Spectrum Archive 1.3.4
  • Spectrum Protect Policies

Functioning policy and expected behaviour​

info

Since /gpfs/tapes is not mounted in the Starlife cluster, all read and write operations must be performed in a Transfer node, using DT Commands

In general, this automatic process is transparent for the user and you can only notice it when you need to access or modify a file that has been migrated. If the file has been migrated, any access to it will be delayed until its content is retrieved from tape.

  • Which files are migrated to tapes and which are not?

Only the files with a size between 1GB and 12TB will be moved (migrated) to tapes from the GPFS disk when no data access and modification have been done for a period of 30 days.

Local Hard Drive​

Every node has a local solid state (SSD) that can be used as a local scratch space to store temporary files during executions of one of your jobs. This space is mounted over /scratch/tmp/$JOBID directory and pointed out by $TMPDIR environment variable. The amount of space within the /scratch filesystem is about 70 GB. All data stored in these local hard drives at the compute nodes will not be available from the login nodes.

You should use the directory referred to by $TMPDIR to save your temporary files during job executions. This directory will automatically be cleaned after the job finishes.

Quotas​

The quotas are the amount of storage available for a user or a groups' users. You can picture it as a small disk readily available to you. A default value is applied to all users and groups and cannot be outgrown.

You can inspect your quota anytime you want using the following command from inside each filesystem: 

bsc_quota

The command provides a readable output for the quota. Check BSC Commands for more information.

If you need more disk space in this filesystem or in any other of the GPFS filesystems, the responsible for your project has to make a request for the extra space needed, specifying the requested space and the reasons why it is needed. For more information or requests you can Contact Us.