Support Knowledge Center

Barcelona Supercomputing Center

Frequently Asked Questions (FAQ)

Table of Contents

  1. General
    1. Backup Policy
    2. How may I identify other members of my team by their username?
    3. Which filesystems do I have available? Which usage is intended?
    4. How can I check how much free disk have I available? What is a quota?
  2. Connections
    1. Why can’t I connect/download/update… from logins to …?
    2. How can I open a GUI in the logins?
    3. ERROR: “/usr/bin/manpath: can’t set the locale; make sure $LC_* and $LANG are correct” or “cannot set LC_CTYPE locale”
  3. Executions
    1. I was copying data/compiling/executing something in the login and the process was killed. Why?
    2. My job failed and I see a message like “OOM Killed…” in the logs. What is this?
    3. My job has been waiting for a long time. How can I see when will it be executed?
    4. My job exits with some error just after starting. What is wrong?
  4. Data Transfer Cluster
    1. What is the Data Transfer Cluster for?
    2. How may I connect to the Data Transfer Cluster?
    3. What is /gpfs/archive? How may I access it?
    4. What are these dtcommands I hear about?
  5. BSC Commands
    1. What are the BSC Commands?
    2. How can I access them?
    3. Which commands are available?
    4. Are they accessible on all machines? When will they be available on […]?
    5. Will there ever be a command for […]?
  6. Queues
    1. Which queues have I available?
    2. Are there any special queues? Which should I use?
    3. How can I use the queues?
    4. How can I know when will a job start?
    5. How can I see if there are any problems with my job?
  7. LSF
    1. What is LSF?
    2. Which machines are using LSF?
    3. How can I check the status of my jobs? What does the status message […] mean?
    4. There are some special requirements for jobs. How should I put them?
  8. SLURM
    1. What is SLURM?
    2. Which machines are using SLURM?
    3. How can I check the status of my jobs? What does the status message […] mean?
  9. Performance Tuning
    1. At […] we saw that using the mpi option […] gives up to […] speedup. Have you seen a similar behavior at […]?
    2. I have found performance problems, what should I do?
  10. Support Team
    1. How/when may I contact Support Team?
    2. Who may access the resources of BSC?
    3. How can I get access to BSC for computing?

General ↩

Backup Policy ↩

The Backup Policy at BSC for the HPC machines is to backup daily the contents of $HOME and /gpfs/projects. For each present file a few versions may be stored. For deleted files, after a few weeks only the last version is kept. If your files are stored elsewhere in the filesystems, there’s no backup available.

How may I identify other members of my team by their username? ↩

To identify other users of your team you may use command ‘finger’. If you specify

finger <username>

It will return the user details for that username including the name.

Which filesystems do I have available? Which usage is intended? ↩

Most of our machines use a shared filesystem (GPFS) across them, so all your data is accessible from your account in all of them. Also, quotas are enforced to limit user data on filesystem. Those machines present the following structure:

Also there is a local hard disk available in each node but not shared nor accessible from anywhere else but the node:

/gpfs/home

This is your home filesystem where you should store your personal and configuration files and is private even for other members of your team.

This filesystem is backed up once a day.

/apps

This is where applications are installed in the machine. Users are not allowed to install software or modify existing installations. Some of the applications may require licenses to be used and its access may be restricted. If your group holds a valid license for any of such software and you can’t access the software, do not hesitate to contact our Support Team.

/gpfs/projects

This filesystem is intended to store the inputs and results of your executions and to share data with other group members.

This filesystem is backed up once a day.

/gpfs/scratch

This filesystem presents the highest performance for distributed read and write and should be used to store the temporary or intermediate files used by your executions. You may copy your inputs to these filesystem for the execution and retrieve the results to /gpfs/projects for analysis.

This filesystem is not backed up and data older than 15 days may be deleted without warning if space is needed.

/gpfs/archive

This filesystem is intended for long-term data storage and is only browsable from Data Transfer. It may be accessed through dtcommands

There is no backup of data. All project’s data may remain there up to a year after end of project to account for project renovations in non-contiguous periods.

/scratch/tmp

This disk is intended for temporary files that don’t need to be recovered, like mesh partitions local to the node or MPI temporary communications between nodes. The data is not recoverable after the job. Do not confound with /gpfs/scratch.

How can I check how much free disk have I available? What is a quota? ↩

All the filesystems you have available have a quota set. These quotas limit the amount of space usable by a user or group of users. We provide some convenience tools to check the usage and limits of quotas depending on the machine you are connected to. Mind that the filesystem is shared so the limits and status is the same across all machines, but may only be checked in a certain way from certain machines.

Nord III and MinoTauro

You can use the command bsc_quota from the bsc_commands bundle which aggregates information about all filesystems in a comprehensive way.

Ultraviolet and Data Transfer

You may use command quota. If you simply state ‘quota’ it will show your user’s quota for the /gpfs/home filesystem. To see your group’s quota usage on /gpfs/projects and /gpfs/scratch you should add flag -g:

quota -g <group>

For /gpfs/archive quota you should issue command dtquota.

Connections ↩

Why can’t I connect/download/update… from logins to …? ↩

For security reasons, our clusters are unable to open outgoing connections to other machines, either internal (other BSC facilities) or external, but they will accept incoming connections. You must upload all needed data to the cluster by yourself and download it the same way.

How can I open a GUI in the logins? ↩

To open GUIs you need to connect with parameter -X in your ssh connection (in Linux/OSX) or with some kind of x11 Forwarding (Windows). Examples:

Linux/OSX

   ssh -X -l username <login.bsc.es>

ERROR: “/usr/bin/manpath: can’t set the locale; make sure $LC_* and $LANG are correct” or “cannot set LC_CTYPE locale” ↩

This error is related to the locale (the language dependent character encoding) of your system being different/incompatible with MareNostrum’s. If you find yourself in this situation, please try the following:

All

    LANG=es_ES.UTF-8 ssh -l username <login.bsc.es>

MacOsX

Some Mac versions have a bug in the terminal that ignores the previous setting and causes this error. You should be able to disable this behaviour by unchecking “Set locale environment variables on startup” in Terminal Settings -> Advanced.

Executions ↩

I was copying data/compiling/executing something in the login and the process was killed. Why? ↩

The logins in our facilities have 5 minutes CPU time limit. This means that any execution requiring more than that is automatically killed.

You may avoid this restriction by using the queues (interactive queue or standard job) to either compile or execute something depending of your interactive needs. To transfer files, you may use Data Transfer commands inside our cluster’s filesystems or connect to our Data Transfer machines, where uploads and downloads aren’t time limited.

My job failed and I see a message like “OOM Killed…” in the logs. What is this? ↩

This is a message from the OS kernel stating that your process was consuming too much memory exceeding the node’s limits and was thus killed.

If you encounter such problem you should try to change how many processes are executed on the same node. It is recommended you contact our Support Team if it’s the first time you try to tweak this settings.

My job has been waiting for a long time. How can I see when will it be executed? ↩

There is no reliable prediction of when a certain job will start executing as the priorities in the queue are composed by several factors and the value may change due to the presence of other jobs. The system is designed so all jobs will be executed eventually, but it may take longer for some of them. The best approach to avoid this to have a huge impact on your project is to submit several jobs, so the waiting times of the different jobs overlap and the grand result is improved execution time due to parellelism.

My job exits with some error just after starting. What is wrong? ↩

This is usually related with some configuration problems of the input or the application itself. Before retrying your submissions, please open an interactive session and manually test a minimal execution to see what the problem is. If nothing turns up or you don’t know how to fix it, please contact our Support Team with detailed explanations of the problems encountered, where are the tests you tried and their results and how to test and reproduce your execution.

If the interactive sessions are too small for your tests, consider a debug queue test job.

Data Transfer Cluster ↩

What is the Data Transfer Cluster for? ↩

The Data Transfer Cluster is a special machine intended to manage the shared filesystems (GPFS) at BSC and provide background data management (compression, archiving copying…) for our users. All accounts at our center are paired with a Data Transfer account.

Unlike the other machine logins, Data Transfer imposes no time limit for transferring files from or to the cluster. It is recommended that big data transfers are done against these machines. Also, the PRACE interconnection network, which interconnects all Tier–0 and most Tier–1 PRACE supercomputing centers, is accessible from these machines (for more information check your machine’s User’s Guide).

How may I connect to the Data Transfer Cluster? ↩

You will need a valid user on our facilities. If you have a valid user for any of our HPC machines you will have a valid user at Data Transfer. Login to one of:

with your credentials.

What is /gpfs/archive? How may I access it? ↩

/gpfs/archive is a long-term storage for our user’s data. As such it is not directly accessible from our HPC machines except via dtcommands.

To access this storage space you will need to login to Data Transfer machines and access it as usual. You should never issue the common data movement commands (cp, mv, tar) to archive but use the dtcommands for performance purposes.

What are these dtcommands I hear about? ↩

The dtcommands are special data movement commands that automatically submit a job to the Data Transfer Cluster so they are not limited by login time. They are also the only effective means to access /gpfs/archive’s data. This commands can be used from the Data Transfer logins and from our other HPC machines.

Some of the commands are briefly presented here. Complete documentation for the full suite of commands is accessible using either ‘man dtcommands’ or ‘man <comamnd_name>’ on any login.

dtcp

Submits a job to copy a path to another path. Accepts and honors all standard cp parameters.

dtmv

Submits a job to move a path to another path, useful for inter-filesystem movements and large renamings. Accepts and honors all standard mv parameters.

dttar

Submits a job to archive/unarchive some files. Accepts and honors all standard tar parameters and options.

dtquota

Submits a job to check the quota available in /gpfs/archive and the other shared filesystems. Form MareNostrum it’s preferred to use bsc_quota.

dtq

Shows the list of jobs submitted that are still pending or running and permits to query them.

dtcancel

Cancels or aborts the job specified.

BSC Commands ↩

What are the BSC Commands? ↩

The BSC Commands is a suite of commands designed to ease user experience on our HPC machines at BSC. These commands intend to ease interaction with parts of the cluster and give some control to users. To interact with the filesystems and further control your data you have dtcommands available.

How can I access them? ↩

You usually will have them loaded on login. To confirm check that:

module list

shows ‘bsc/current’ as loaded. If it isn’t, you can load them with:

module load bsc

This will only work on some machines.

Which commands are available? ↩

Here is a short description of the available commands. Full documentation is available through manpages:

man bsc
man bsc_commands

bsc_acct

Displays accounting information about the project’s allocation usage.

bsc_jobcheck

Simulates the submission of a job and reports the resources it would use and its distribution if submitted.

bsc_jobs

Lists all pending and running jobs for user and other group members.

bsc_load

Instantaneous load metrics about running job on all its nodes.

bsc_queues

Lists available queues and queue limits for the user.

bsc_quota

Comprehensive summary of quota usage for user and group on all filesystems.

Are they accessible on all machines? When will they be available on […]? ↩

Right now the commands are only available on Nord III. There is a Work In Progress to port the commands to all our machines. Some commands are not portable to other machines because they make no sense.

Machine bsc_acct bsc_jobcheck bsc_jobs bsc_load bsc_queues bsc_quota
Nord DONE DONE DONE DONE DONE DONE
MinoTauro TESTING - WIP DONE DONE DONE
UltraViolet NEVER - - - TESTING TESTING
Data Transfer NEVER NEVER dtq NEVER NEVER dtquota

On the other hand, dtcommands are available on all machines right now.

Will there ever be a command for […]? ↩

If you think there may be need for additional commands or it would be better to change the behaviour of some command, please contact our Support Team. Please also check dtcommands for other useful commands.

Queues ↩

Which queues have I available? ↩

Nord

If you want to check which queues are available to you, you may issue the command bsc_queues from the bsc_commands bundle on any opened session in the machine. This command lists all the queues your user has access to and their limits in number of CPUs and in time.

Other machines

Right now there is no standard way of accessing this information on other machines. You can check the status of bsc_commands porting.

Are there any special queues? Which should I use? ↩

Most executions rely on the default queue that is automatically selected for each user depending on their membership and consumption. Otherwise there are some special property queues that must be specified to use:

debug

This is a small queue (only a few nodes, check limits with bsc_queues) useful for debugging applications and testing the pipeline for your production jobs. It’s short time and small number of nodes available makes it unsuitable for production jobs but it’s usually easier to run your tests there.

Note: Penalties may be incurred if abusive usage of this queue is detected.

interactive

This queue allows you to run a shell on a compute node. This is useful for large compilations and interactive preparation of inputs or interactive result analysis that may be unsuitable for the login nodes. Mind that there are still time limits specified when submitting.

x11

This queue allows to run interactive graphical jobs, meaning that you will be able to open a GUI for interacting with your application. Mind you will need to connect to the cluster with x11 Forwarding enabled How can I open a GUI in the logins?.

Note: Penalties may be incurred if abusive usage of this queue is detected.

sequential

This queue is automatically selected for those jobs that request a node’s resources or less. This queue should NEVER be requested by a jobscript as it may increase waiting times and cause havoc in the queue system.

How can I use the queues? ↩

Our machines use different queue systems so please refer to the machine’s User Guide for specific instructions on submitting jobs and writing jobscripts.

How can I know when will a job start? ↩

There is no way to predict the time or date when a job will start. All jobs are continuously rescheduled based on their requirements, the cluster’s status and the other jobs queued. This makes the priority of a job remain predictable but the priority threshold for execution be in continuous flow.

A rule of thumb for executions would be: * Big executions have higher priority * Sequential queue has the least priority * Small and short jobs may benefit form backfilling

If you think the waiting times for your job are abnormal you may check if there are any problems with your job.

How can I see if there are any problems with my job? ↩

Right now, you can only ask the scheduler about the reason your job is still pending. This reason may point to some job resource request problems, excessive machine load or administrative restrictions. In any case the messages and methods are scheduler dependent. Please check your machine’s Users Guide to determine which scheduler affects you.

LSF (Nord III)

Please check this.

Slurm

Please check this.

LSF ↩

What is LSF? ↩

LSF is a job scheduler for some of our HPC machines. This is the program used to interpret how many resources a job requires, control how many resources are being used and, where and when should a new job be allocated. It also controls the execution state of the jobs, the accounting and interfaces with the MPI libraries to distribute the resources among the allocation.

If you need more information please check the User Guide for your machine.

Which machines are using LSF? ↩

Right now only Nord III uses LSF as queue manager.

How can I check the status of my jobs? What does the status message […] mean? ↩

Use bjobs or bsc_jobs and also check the User’s Guide for your machine.

LSF reports several state messages for the jobs. Some imply critical issues with the job that will prevent it from running while others are just informative messages. Here we list a few of them with their meaning. If your message doesn’t appear, do not hesitate to contact our Support Team.

PEND

The job is scheduled to run and is waiting for a execution slot be freed.

RUN

The job is running.

PSUSP

The job is suspended. Before it started to run it was put on hold using bstop (won’t be scheduled for execution). You can resume it with bresume or remove it with bkill.

There are some special requirements for jobs. How should I put them? ↩

Reservations

Sometimes and specially for training hands-on sessions some nodes may be reserved to use exclusively by certain group. To use the reserved nodes, you will need to know the reservation name and add the following parameter to your jobs:

    #BSUB -U reserve_name

Threads

Check the job examples on your machine’s User Guide.

Job’s power consumption

We provide power consumption metrics for Nord’s jobs. Nonetheless, be aware it is considered Beta state and may result unreliable. Also, it is not available everywhere, note that:

If you have any doubts or issues on using this feature please contact our Support Team.

SLURM ↩

What is SLURM? ↩

SLURM stands for Simple Linux Utility for Resource Management and it’s used as job scheduler. This program interprets the amount of resources a job requires and ontrols how many resources are available for new jobs. It also controls the execution state of the jobs and interfaces with the MPI libraries to distribute the resources among the allocation.

If you need more information please check the User Guide for your machine.

Which machines are using SLURM? ↩

Right now both MinoTauro and UltraViolet (UV100) are using SLURM as their queue manager.

How can I check the status of my jobs? What does the status message […] mean? ↩

Use mnq and check the User’s Guide for your machine.

SLURM reports several state messages for the jobs. Some imply critical issues with the job that will prevent it from running while others are just informative messages. Here we list a few of them with their meaning. If your message doesn’ appear in this list, do not hesitate to contact our Support Team.

AssociationJobLimit

There is a limit on the amount of jobs a user or group may have running at the same time. This message means your job will be kept back because your group has filled all your running slots and it must wait until one of the running jobs is over before being able to start execution.

Priority

The job doesn’t have enough priority to start requesting resources for execution. As time passes and higher priority jobs get executed this job will become of the highest priority and will start reserving resources and eventually being executed.

PEND

The job is scheduled to run but other jobs have higher priority and it’s waiting for them to start execution.

RUN

The job is running.

Resources

There are not enough resources available to satisfy the job’s requirements. The job will wait until enough resources become available. Usually means the machine is full.

ReqNodeNotAvail

Some of the nodes requested are not available. They may be reserved or in maintenance, among other reasons. This message may appear when using certain partitions or reservations. You may contact with our Support Team for further information.

Performance Tuning ↩

At […] we saw that using the mpi option […] gives up to […] speedup. Have you seen a similar behavior at […]? ↩

We have seen that the tuning options of our different MPI implementations have very different outcomes depending on the application that is run and, sometimes, even on the input. We recommend that you benchmark different options to see which is best for you.

Our general recommendations are:

I have found performance problems, what should I do? ↩

First, you should find a method to reproduce the problem and confirm after some tests that it is reproducible. After that, you should provide all relevant information to the Support Team by e-mail together with instructions on how to reproduce your tests. Support Team will investigate the issue and contact you as soon as possible.

Support Team ↩

How/when may I contact Support Team? ↩

You may send an e-mail any time and it will be answered on the next working day on office hours (9:00 - 18:00 CET). Bank holidays correspond to Barcelona’s. Depending on your Access Project:

Who may access the resources of BSC? ↩

Scientific access to BSC’s HPC resources is granted through some national and international research projects:

How can I get access to BSC for computing? ↩

There are periodic proposal submission deadlines for new and continuing projects 3 times a year. Visit each Access Project’s website to check when is the next submission deadline.