bjobs

• X: The real run time has exceeded the configured run time.
• L: A run limit exists but job does not have a configured run time.
• E: A configured run time exists and has not been exceeded.

-A

Displays summarized information about job arrays. If you specify job arrays with the job array ID, and also specify -A, do not include the index list with the job array ID.

You can use -w to show the full array specification, if necessary.

-a

Displays information about jobs in all states, including finished jobs that finished recently, within an interval specified by CLEAN_PERIOD in lsb.params (the default period is 1 hour).

Use -a with -x option to display all jobs that have triggered a job exception (overrun, underrun, idle).

-aff

Displays information about jobs with CPU and memory affinity resource requirements for each task in the job. If the job is pending, the requested affinity resources are displayed. For running jobs, the effective and combined affinity resource allocation decision made by LSF is also displayed, along with a table headed AFFINITY that shows detailed memory and CPU binding information for each task, one line for each allocated processor unit. For finished jobs (EXIT or DONE state), the affinity requirements for the job, and the effective and combined affinity resource requirement details are displayed.

Use bhist -l -aff to show the actual affinity resource allocation for finished jobs.

Use only with the -l or -UF option.

-aps

Displays absolute priority scheduling (APS) information for pending jobs in a queue with APS_PRIORITY enabled. The APS value is calculated based on the current scheduling cycle, so jobs are not guaranteed to be dispatched in this order.

Pending jobs are ordered by APS value. Jobs with system APS values are listed first, from highest to lowest APS value. Jobs with calculated APS values are listed next ordered from high to low value. Finally, jobs not in an APS queue are listed. Jobs with equal APS values are listed in order of submission time. APS values of jobs not in an APS queue are shown with a dash (-).

If queues are configured with the same priority, bjobs -aps may not show jobs in the correct expected dispatch order. Jobs may be dispatched in the order the queues are configured in lsb.queues. You should avoid configuring queues with the same priority.

For resizable jobs, -aps displays the latest APS information for running jobs with active resize allocation requests. LSF handles the dynamic priority for running jobs with active resize requests. The displayed job priority can change from time to time.

-cname

In LSF Advanced Edition, includes the cluster name for execution cluster hosts in output.

% bjobs -l -cname
Job <1>, User <lsfuser>, Project <default>, Status <RUN>, Queue <queue1>, 
Command <sleep 1234>
Mon Nov 29 14:08:35: Submitted from host <hostA>, CWD </home/lsfuser >, 
                     Re-runnable;
Mon Nov 29 14:08:38: Job <1> forwarded to cluster <cluster3>;
Mon Nov 29 14:08:44: Started on <hostC@cluster3>, Execution Home 
                     </home/lsfuser>, Execution CWD </home/lsfuser>;
Mon Nov 29 14:08:46: Resource usage collected.
 MEM: 2 Mbytes;  SWAP: 32 Mbytes;  NTHREAD: 1
 PGID: 6395;  PIDs: 6395
 
SCHEDULING PARAMETERS:
           r15s r1m  r15m ut pg io ls it tmp swp mem
 loadSched -    -    -    -  -  -  -  -   -  -   -
 loadStop  -    -    -    -  -  -  -  -   -  -   -
 ...

-d

Displays information about jobs that finished recently, within an interval specified by CLEAN_PERIOD in lsb.params (the default period is 1 hour).

-fwd

In MultiCluster job forwarding mode, filters output to display information on forwarded jobs, including the forwarded time and the name of the cluster to which the job was forwarded. -fwd can be used with other options to further filter the results. For example, bjobs -fwd -r displays only forwarded running jobs.

% bjobs -fwd
JOBID USER    STAT QUEUE  EXEC_HOST JOB_NAME   CLUSTER  FORWARD_TIME
123   lsfuser RUN  queue1 hostC     sleep 1234 cluster3 Nov 29 14:08

The following bjobs options cannot be used together with -fwd: -A, -d, -sla, -ss, -x.

To use -x to see exceptions on the execution cluster, use bjobs -m execution_cluster -x.

-l

Long format. Displays detailed information for each job in a multiline format.

The -l option displays the following additional information: project name, job command, current working directory on the submission host, initial checkpoint period, checkpoint directory, migration threshold, pending and suspending reasons, job status, resource usage, resource usage limits information, runtime resource usage information on the execution hosts, and job description.

If the job was submitted with bsub -K, the -l option displays Synchronous Execution.

Use bjobs -A -l to display detailed information for job arrays including job array job limit (% job_limit) if set.

Use bjobs -ss -l to display detailed information for session scheduler jobs.

If JOB_IDLE is configured in the queue, use bjobs -l to display job idle exception information.

If you submitted your job with the -U option to use advance reservations created with the brsvadd command, bjobs -l shows the reservation ID used by the job.

If LSF_HPC_EXTENSIONS="SHORT_PIDLIST" is specified in lsf.conf, the output from bjobs is shortened to display only the first PID and a count of the process group IDs (PGIDs) and process IDs for the job. Without SHORT_PIDLIST, all of the process IDs (PIDs) for a job are displayed.

If LSF_HPC_EXTENSIONS="HOST_RUSAGE" is specified in lsf.conf, the output from bjobs -l reports the correct rusage based on each host’s usage and the total rusage being charged to the execution host.

If you submitted a job with multiple resource requirement strings using the bsub -R option for the order, same, rusage, and select sections, bjobs -l displays a single, merged resource requirement string for those sections, as if they were submitted using a single -R.

If you submitted a job using the OR (||) expression to specify alternative resources, this option displays the Execution rusage string with which the job runs.

For resizable jobs, the -l option displays active pending resize allocation requests, and the latest job priority for running jobs with active pending resize requests.

For jobs with user-based fairshare scheduling, displays the charging SAAP (share attribute account path).

For jobs submitted to an absolute priority scheduling (APS) queue, -l shows the ADMIN factor value and the system APS value if they have been set by the administrator for the job.

For jobs submitted with SSH X11 forwarding, displays that the job was submitted in SSH X11 forwarding mode as well as the SSH command submitted (set in LSB_SSH_XFORWARD_CMD in lsf.conf.)

If the job was auto-attached to a guarantee SLA, -l displays the auto-attached SLA name.

Specified CWD shows the value of the bsub -cwd option or the value of LSB_JOB_CWD. The CWD path with pattern values is displayed. CWD is the submission directory where bsub ran. If specified CWD was not defined, this field is not shown. The execution CWD with pattern values is always shown.

If the job was submitted with an energy policy, to automatically select a CPU frequency, -l will show the Combined CPU frequency (the CPU frequency selected for the job based on the energy policy tag, energy policy and threshold file). If the job was submitted with a user defined CPU frequency (using bsub –freq), -l will show the Specified CPU frequency for the job.

-noheader

Removes the column headings from the output.

When specified, the bjobs displays the values of the fields without displaying the names of the fields. This is useful for script parsing, when column headings are not necessary.

This option applies to output for the bjobs command with no options, and to output for all bjobs options with short form output except for -aff, -l, -UF, -N, -h, and -V.

-o "field_name[:[-][output_width]] ... [delimiter='character']"

Sets the customized output format.

• Specify which bjobs fields (or aliases instead of the full field names), in which order, and with what width to display.
• Specify only the bjobs field name or alias to set its output to unlimited width and left justification.
• Specify the colon (:) without a width to set the output width to the recommended width for that field.
• Specify the colon (:) with a width to set the maximum number of characters to display for the field. When its value exceeds this width, bjobs truncates the output as follows:
  • For the JOB_NAME field, bjobs removes the header characters and replaces them with an asterisk (*)
  • For other fields, bjobs truncates the ending characters
• Specify a hyphen (-) to set right justification when displaying the output for the specific field. If not specified, the default is to set left justification when displaying output for a field.
• Use delimiter= to set the delimiting character to display between different headers and fields. This must be a single character. By default, the delimiter is a space.

The following alternate method of using bjobs -o is recommended for special delimiter characters in a csh environment (for example, $):

bjobs ... -o 'field_name[:[-][output_width]] ... [delimiter="character"]'

The -o option only applies to output for certain bjobs options, as follows:

• This option applies to output for the bjobs command with no options, and for bjobs options with short form output that filter information, including the following: -a, -app, -cname, -d, -g, -G, -J, -Jd, -Lp, -m, -P, -q, -r, -sla, -u, -x, -X.
• This option applies to output for bjobs options that use a modified format and filter information, including the following: -fwd, -N, -p, -s.
• This option does not apply to output for bjobs options that use a modified format, including the following: -A, -aff, -aps, -l, -UF, -ss, -sum, -UF, -w, -W, -WF, -WL, -WP.

The bjobs -o option overrides the LSB_BJOBS_FORMAT environment variable, which overrides the LSB_BJOBS_FORMAT setting in lsf.conf.

The following are the field names used to specify the bjobs fields to display, recommended width, aliases you can use instead of field names, and units of measurement for the displayed field:

1. Common information:
   1. jobid
      • width: 7
      • alias: id
   2. stat
      • width: 5
   3. user
      • width: 7
   4. queue
      • width: 10
   5. job_name
      • width: 10
      • alias: name
   6. job_description
      • width: 17
      • alias: description
   7. proj_name
      • width: 11
      • aliases: proj and project
   8. application
      • width: 13
      • alias: app
   9. service_class
      • width: 13
      • alias: sla
   10. job_group
       • width: 10
       • alias: group
   11. job_priority
       • width: 12
       • alias: priority
   12. dependency
       • width: 15
2. Command information:
   1. command
      • width: 15
      • alias: cmd
   2. pre_exec_command
      • width: 16
      • alias: pre_cmd
   3. post_exec_command
      • width: 17
      • alias: post_cmd
   4. resize_notification_command
      • width: 27
      • alias: resize_cmd
   5. pids
      • width: 20
   6. exit_code
      • width: 10
   7. exit_reason
      • width: 50
3. Host information:
   1. from_host
      • width: 11
   2. first_host
      • width: 11
   3. exec_host
      • width: 11
   4. nexec_host
      • width: 10
      Note: If the allocated host group or compute unit is condensed, this field does not display the real number of hosts. Use bjobs -X -o to view the real number of hosts in these situations.
4. Time information:
   1. submit_time
      • width: 15
   2. start_time
      • width: 15
   3. estimated_start_time
      • width: 20
      • alias: estart_time
   4. specified_start_time
      • width: 20
      • alias: sstart_time
   5. specified_terminate_time
      • width: 24
      • alias: sterminate_time
   6. time_left
      • width: 11
      • unit of measurement: seconds
   7. finish_time
      • width: 16
   8. %complete
      • width: 11
   9. warning_action
      • width: 15
      • alias: warn_act
   10. action_warning_time
       • width: 19
       • alias: warn_time
5. CPU information:
   1. cpu_used
      • width: 10
   2. run_time
      • width: 15
      • unit of measurement: seconds
   3. idle_factor
      • width: 11
   4. exception_status
      • width: 16
      • alias: except_stat
   5. slots
      • width: 5
6. Memory information:
   1. mem
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
   2. max_mem
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
   3. avg_mem
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
   4. memlimit
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
   5. swap
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
   6. swaplimit
      • width: 10
      • unit of measurement: LSF_UNIT_FOR_LIMITS in lsf.conf (KB by default)
7. Resource requirement information:
   1. min_req_proc
      • width: 12
   2. max_req_proc
      • width: 12
   3. effecive_resreq
      • width: 17
      • alias: eresreq
   4. network_req
      • width: 15
8. Resource limits:
   1. filelimit
      • width: 10
   2. corelimit
      • width: 10
   3. stacklimit
      • width: 10
   4. processlimit
      • width: 12
9. File information:
   1. input_file
      • width: 10
   2. output_file
      • width: 11
   3. error_file
      • width: 10
10. Directory information:
    1. output_dir
       • width: 15
    2. sub_cwd
       • width: 10
    3. exec_home
       • width: 10
    4. exec_cwd
       • width: 10
11. MultiCluster information
    1. forward_cluster
       • width: 15
       • alias: fwd_cluster
    2. forward_time
       • width: 15
       • alias: fwd_time

Field names and aliases are not case sensitive. Valid values for the output width are any positive integer between 1 and 4096. If the jobid field is defined with no output width and LSB_JOBID_DISP_LENGTH is defined in lsf.conf, the LSB_JOBID_DISP_LENGTH value is used for the output width. If jobid is defined with a specified output width, the specified output width overrides the LSB_JOBID_DISP_LENGTH value.

For example,

bjobs -o "jobid stat: queue:- project:10 application:-6 delimiter='^'" 123

This command displays the following fields for a job with the job ID 123:

• JOBID with unlimited width and left justified. If LSB_JOBID_DISP_LENGTH is specified, that value is used for the output width instead.
• STAT with a maximum width of five characters (which is the recommended width) and left justified.
• QUEUE with a maximum width of ten characters (which is the recommended width) and right justified.
• PROJECT with a maximum width of ten characters and left justified.
• APPLICATION with a maximum width of six characters and right justified.
• The ^ character is displayed between different headers and fields.

-p

Displays pending jobs, together with the pending reasons that caused each job not to be dispatched during the last dispatch turn. The pending reason shows the number of hosts for that reason, or names the hosts if -l is also specified.

With MultiCluster, -l shows the names of hosts in the local cluster.

Each pending reason is associated with one or more hosts and it states the cause why these hosts are not allocated to run the job. In situations where the job requests specific hosts (using bsub -m), users may see reasons for unrelated hosts also being displayed, together with the reasons associated with the requested hosts.

In case of host-based pre-execution failure, pending reasons will be displayed.

The life cycle of a pending reason ends after the time indicated by PEND_REASON_UPDATE_INTERVAL in lsb.params.

When the job slot limit is reached for a job array (bsub -J "jobArray[indexList]%job_slot_limit") the following message is displayed:

The job array has reached its job slot limit.

-r

Displays running jobs.

-s

Displays suspended jobs, together with the suspending reason that caused each job to become suspended.

The suspending reason may not remain the same while the job stays suspended. For example, a job may have been suspended due to the paging rate, but after the paging rate dropped another load index could prevent the job from being resumed. The suspending reason is updated according to the load index. The reasons could be as old as the time interval specified by SBD_SLEEP_TIME in lsb.params. The reasons shown may not reflect the current load situation.

-ss

Displays summary information for session scheduler tasks including the job ID, the owner, the job name (useful for job arrays), the total number of tasks, the state of pending, done, running, and exited session scheduler tasks.

-ss can only display the summary information for session scheduler tasks when the job session has started . -ss cannot display the information while session scheduler job is still pending.

The frequency of the updates of this information is based on the parameters SSCHED_UPDATE_SUMMARY_INTERVAL and SSCHED_UPDATE_SUMMARY_BY_TASK.

The following options cannot be used with -ss: -A, -fwd, -W, -WL, -WF, -WP, -N, -aps.

-sum

Displays summary information about unfinished jobs. bjobs -sum displays the count of job slots in the following states: running (RUN), system suspended (SSUSP), user suspended (USUSP), pending (PEND), forwarded to remote clusters and pending (FWD_PEND), and UNKNOWN.

bjobs -sum displays the job slot count only for the user’s own jobs.

% bjobs -sum
RUN        SSUSP       USUSP      UNKNOWN    PEND       FWD_PEND
123        456         789        5          5          3

Use -sum with other options (like -m, -P, -q, and -u) to filter the results. For example, bjobs -sum -u user1 displays job slot counts just for user user1.

% bjobs -sum -u user1
RUN        SSUSP       USUSP      UNKNOWN    PEND       FWD_PEND
20         10          10         0           5          0

-UF

Displays unformatted job detail information. This makes it easy to write scripts for parsing keywords on bjobs. The results of this option have no wide control for the output. Each line starts from the beginning of the line. Information for SCHEDULING PARAMETERS and PENDING REASONS remain formatted. The rusage message lines ending without any separator have a semicolon added to separate their different parts. The first line and all lines starting with the time stamp are displayed unformatted in a single line. There is no line length and format control. For example:

SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

-W

Provides resource usage information for: PROJ_NAME, CPU_USED, MEM, SWAP, PIDS, START_TIME, FINISH_TIME. Displays jobs that belong to you only if you are not logged in as an administrator.

-WF

Displays an estimated finish time for running or pending jobs. For done or exited jobs, displays the actual finish time.

-WL

Displays the estimated remaining run time of jobs.

-WP

Displays the current estimated completion percentage of jobs.

-w

Wide format. Displays job information without truncating fields.

-X

Displays uncondensed output for host groups and compute units.

-x

Displays unfinished jobs that have triggered a job exception (overrun, underrun, idle, runtime_est_exceeded). Use with the -l option to show the actual exception status. Use with -a to display all jobs that have triggered a job exception.

-app application_profile_name

Displays information about jobs submitted to the specified application profile. You must specify an existing application profile.

-G user_group

Only displays jobs associated with a user group submitted with bsub -G for the specified user group. The –G option does not display jobs from subgroups within the specified user group. Jobs associated with the user group at submission are displayed, even if they are later switched to a different user group.

The -G option cannot be used together with the -u option. You can only specify a user group name. The keyword all is not supported for -G.

-g job_group_name

Displays information about jobs attached to the job group specified by job_group_name. For example:

bjobs -g /risk_group
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
113     user1   PEND  normal     hostA                   myjob     Jun 17 16:15
111     user2   RUN   normal     hostA       hostA       myjob     Jun 14 15:13
110     user1   RUN   normal     hostB       hostA       myjob     Jun 12 05:03
104     user3   RUN   normal     hostA       hostC       myjob     Jun 11 13:18

Use -g with -sla to display job groups attached to a time-based service class. Once a job group is attached to a time-based service class, all jobs submitted to that group are subject to the SLA.

bjobs -l with -g displays the full path to the group to which a job is attached. For example:

bjobs -l -g /risk_group

Job <101>, User <user1>, Project <default>, Job Group </risk_group>, Status 
<RUN>, Queue <normal>, Command <myjob>

Tue Jun 17 16:21:49 2009: Submitted from host <hostA>, CWD </home/user1;

Tue Jun 17 16:22:01 2009: Started on <hostA>;

...

-J job_name

Displays information about the specified jobs or job arrays. Only displays jobs that were submitted by the user running this command.

The job name can be up to 4094 characters long. Job names are not unique.

The wildcard character (*) can be used anywhere within a job name, but cannot appear within array indices. For example job* returns jobA and jobarray[1], *AAA*[1] returns the first element in all job arrays with names containing AAA, however job1[*] will not return anything since the wildcard is within the array index.

-Jd job_description

Displays information about the specified jobs or job arrays. Only displays jobs that were submitted by the user running this command.

The job description can be up to 4094 characters long. Job descriptions are not unique.

The wildcard character (*) can be used anywhere within a job description.

-Lp ls_project_name

Displays jobs that belong to the specified License Scheduler project.

-m host_name ... | -m host_group ... | -m cluster_name ...

Only displays jobs dispatched to the specified hosts. To see the available hosts, use bhosts.

If a host group or compute unit is specified, displays jobs dispatched to all hosts in the group. To determine the available host groups, use bmgroup. To determine the available compute units, use bmgroup -cu.

With MultiCluster, displays jobs in the specified cluster. If a remote cluster name is specified, you see the remote job ID, even if the execution host belongs to the local cluster. To determine the available clusters, use bclusters.

-N host_name | -N host_model | -N cpu_factor

Displays information about done and exited jobs, also displays the normalized CPU time consumed by the job. Normalizes using the CPU factor specified, or the CPU factor of the host or host model specified.

Use with -p, -r, and -s to show information about pending, running, and suspended jobs along with done and exited jobs.

-P project_name

Only displays jobs that belong to the specified project.

-q queue_name

Only displays jobs in the specified queue.

The command bqueues returns a list of queues configured in the system, and information about the configurations of these queues.

In MultiCluster, you cannot specify remote queues.

-sla service_class_name

Displays jobs belonging to the specified service class.

bjobs also displays information about jobs assigned to a default SLA configured with ENABLE_DEFAULT_EGO_SLA in lsb.params.

Use -sla with -g to display job groups attached to a time-based service class. Once a job group is attached to a service class, all jobs submitted to that group are subject to the SLA.

Use bsla to display the configuration properties of service classes configured in lsb.serviceclasses, the default SLA configured in lsb.params, and dynamic information about the state of each service class.

-u user_name... | -u user_group... | -u all

Only displays jobs that have been submitted by the specified users or user groups. The keyword all specifies all users. To specify a Windows user account, include the domain name in uppercase letters and use a single backslash (DOMAIN_NAME\user_name) in a Windows command line or a double backslash (DOMAIN_NAME\\user_name) in a UNIX command line.

The -u option cannot be used with the -G option.

job_ID | "job_ID[index]"

Displays information about the specified jobs or job arrays.

If you use -A, specify job array IDs without the index list.

In MultiCluster job forwarding mode, you can use the local job ID and cluster name to retrieve the job details from the remote cluster. The query syntax is:

bjobs submission_job_id@submission_cluster_name

For job arrays, the query syntax is:

bjobs "submission_job_id[index]"@submission_cluster_name

The advantage of using submission_job_id@submission_cluster_name instead of bjobs -l job_id is that you can use submission_job_id@submission_cluster_name as an alias to query a local job in the execution cluster without knowing the local job ID in the execution cluster. The bjobs output is identical no matter which job ID you use (local job ID or submission_job_id@submission_cluster_name).

You can use bjobs 0 to find all jobs in your local cluster, but bjobs 0@submission_cluster_name is not supported.

-h

Prints command usage to stderr and exits.

-help [command_option | command_sub_option]

Displays the description of the specified command option or sub-option to stderr and exits.

Run bjobs -help without a command option name to display the bjobs command description.

For example:

bjobs -help -app
      -app application_profile_name
            Displays information about jobs submitted 
            to the specified application profile. 
            You must specify an existing application 
            profile.

-V

Prints LSF release version to stderr and exits.

JOBID

The job ID that LSF assigned to the job.

USER

The user who submitted the job.

STAT

The current status of the job (see JOB STATUS below).

QUEUE

The name of the job queue to which the job belongs. If the queue to which the job belongs has been removed from the configuration, the queue name is displayed as lost_and_found. Use bhist to get the original queue name. Jobs in the lost_and_found queue remain pending until they are switched with the bswitch command into another queue.

In a MultiCluster resource leasing environment, jobs scheduled by the consumer cluster display the remote queue name in the format queue_name@cluster_name. By default, this field truncates at 10 characters, so you might not see the cluster name unless you use -w or -l.

FROM_HOST

The name of the host from which the job was submitted.

With MultiCluster, if the host is in a remote cluster, the cluster name and remote job ID are appended to the host name, in the format host_name@cluster_name:job_ID. By default, this field truncates at 11 characters; you might not see the cluster name and job ID unless you use -w or -l.

EXEC_HOST

The name of one or more hosts on which the job is executing (this field is empty if the job has not been dispatched). If the host on which the job is running has been removed from the configuration, the host name is displayed as lost_and_found. Use bhist to get the original host name.

If the host is part of a condensed host group or compute unit, the host name is displayed as the name of the condensed group.

If you configure a host to belong to more than one condensed host groups using wildcards, bjobs can display any of the host groups as execution host name.

JOB_NAME

The job name assigned by the user, or the command string assigned by default at job submission with bsub. If the job name is too long to fit in this field, then only the latter part of the job name is displayed.

The displayed job name or job command can contain up to 4094 characters for UNIX, or up to 255 characters for Windows.

SUBMIT_TIME

The submission time of the job.

Project

The project the job was submitted from.

Application Profile

The application profile the job was submitted to.

Command

The job command.

CWD

The current working directory on the submission host.

Execution CWD

The actual CWD used when job runs.

Initial checkpoint period

The initial checkpoint period specified at the job level, by bsub -k, or in an application profile with CHKPNT_INITPERIOD.

Checkpoint period

The checkpoint period specified at the job level, by bsub -k, in the queue with CHKPNT, or in an application profile with CHKPNT_PERIOD.

Checkpoint directory

The checkpoint directory specified at the job level, by bsub -k, in the queue with CHKPNT, or in an application profile with CHKPNT_DIR.

Migration threshold

The migration threshold specified at the job level, by bsub -mig.

Post-execute Command

The post-execution command specified at the job-level, by bsub -Ep.

PENDING REASONS

The reason the job is in the PEND or PSUSP state. The names of the hosts associated with each reason are displayed when both -p and -l options are specified.

SUSPENDING REASONS

The reason the job is in the USUSP or SSUSP state.

loadSched

The load scheduling thresholds for the job.

loadStop

The load suspending thresholds for the job.

JOB STATUS

Possible values for the status of a job include:

PEND

The job is pending. That is, it has not yet been started.

PROV

The job has been dispatched to a power-saved host that is waking up. Before the job can be sent to the sbatchd, it is in a PROV state.

PSUSP

The job has been suspended, either by its owner or the LSF administrator, while pending.

RUN

The job is currently running.

USUSP

The job has been suspended, either by its owner or the LSF administrator, while running.

SSUSP

The job has been suspended by LSF. The job has been suspended by LSF due to either of the following two causes:

• The load conditions on the execution host or hosts have exceeded a threshold according to the loadStop vector defined for the host or queue.
• The run window of the job’s queue is closed. See bqueues(1), bhosts(1), and lsb.queues(5).

DONE

The job has terminated with status of 0.

EXIT

The job has terminated with a non-zero status – it may have been aborted due to an error in its execution, or killed by its owner or the LSF administrator.

For example, exit code 131 means that the job exceeded a configured resource usage limit and LSF killed the job.

UNKWN

mbatchd has lost contact with the sbatchd on the host on which the job runs.

WAIT

For jobs submitted to a chunk job queue, members of a chunk job that are waiting to run.

ZOMBI

A job becomes ZOMBI if:

• A non-rerunnable job is killed by bkill while the sbatchd on the execution host is unreachable and the job is shown as UNKWN.
• The host on which a rerunnable job is running is unavailable and the job has been requeued by LSF with a new job ID, as if the job were submitted as a new job.
• After the execution host becomes available, LSF tries to kill the ZOMBI job. Upon successful termination of the ZOMBI job, the job’s status is changed to EXIT.

  With MultiCluster, when a job running on a remote execution cluster becomes a ZOMBI job, the execution cluster treats the job the same way as local ZOMBI jobs. In addition, it notifies the submission cluster that the job is in ZOMBI state and the submission cluster requeues the job.

RUNTIME

Estimated run time for the job, specified by bsub -We or bmod -We, -We+, -Wep.

The following information is displayed when running bjobs -WL, -WF, or -WP.

TIME_LEFT

The estimated run time that the job has remaining. Along with the time if applicable, one of the following symbols may also display.

• E: The job has an estimated run time that has not been exceeded.
• L: The job has a hard run time limit specified but either has no estimated run time or the estimated run time is more than the hard run time limit.
• X: The job has exceeded its estimated run time and the time displayed is the time remaining until the job reaches its hard run time limit.
• A dash indicates that the job has no estimated run time and no run limit, or that it has exceeded its run time but does not have a hard limit and therefore runs until completion.

If there is less than a minute remaining, 0:0 displays.

FINISH_TIME

The estimated finish time of the job. For done/exited jobs, this is the actual finish time. For running jobs, the finish time is the start time plus the estimated run time (where set and not exceeded) or the start time plus the hard run limit.

• E: The job has an estimated run time that has not been exceeded.
• L: The job has a hard run time limit specified but either has no estimated run time or the estimated run time is more than the hard run time limit.
• X: The job has exceeded its estimated run time and had no hard run time limit set. The finish time displayed is the estimated run time remaining plus the start time.
• A dash indicates that the pending, suspended, or job with no run limit has no estimated finish time.

%COMPLETE

The estimated completion percentage of the job.

• E: The job has an estimated run time that has not been exceeded.
• L: The job has a hard run time limit specified but either has no estimated run time or the estimated run time is more than the hard run time limit.
• X: The job has exceeded its estimated run time and had no hard run time limit set.
• A dash indicates that the jobs is pending, or that it is running or suspended, but has no run time limit specified.

Note: For jobs in the state UNKNOWN, the job run time estimate is based on internal counting by the job’s mbatchd.

RESOURCE USAGE

For the MultiCluster job forwarding model, this information is not shown if MultiCluster resource usage updating is disabled. Use LSF_HPC_EXTENSIONS="HOST_RUSAGE" in lsf.conf to specify host-based resource usage.

The values for the current usage of a job include:

HOST

For host-based resource usage, specifies the host.

CPU time

Cumulative total CPU time in seconds of all processes in a job. For host-based resource usage, the cumulative total CPU time in seconds of all processes in a job running on a host.

IDLE_FACTOR

Job idle information (CPU time/runtime) if JOB_IDLE is configured in the queue, and the job has triggered an idle exception.

MEM

Total resident memory usage of all processes in a job. For host-based resource usage, the total resident memory usage of all processes in a job running on a host. The sum of host-based rusage may not equal the total job rusage, since total job rusage is the maximum historical value.

By default, memory usage is shown in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for display (MB, GB, TB, PB, or EB).

SWAP

Total virtual memory usage of all processes in a job. For host-based resource usage, the total virtual memory usage of all processes in a job running on a host. The sum of host-based rusage may not equal the total job rusage, since total job rusage is the maximum historical value.

By default, swap space is shown in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for display (MB, GB, TB, PB, or EB).

NTHREAD

Number of currently active threads of a job.

PGID

Currently active process group ID in a job. For host-based resource usage, the currently active process group ID in a job running on a host.

PIDs

Currently active processes in a job. For host-based resource usage, the currently active active processes in a job running on a host.

RESOURCE LIMITS

The hard resource usage limits that are imposed on the jobs in the queue (see getrlimit(2) and lsb.queues(5)). These limits are imposed on a per-job and a per-process basis.

The possible per-job resource usage limits are:

• CPULIMIT
• PROCLIMIT
• MEMLIMIT
• SWAPLIMIT
• PROCESSLIMIT
• THREADLIMIT
• OPENFILELIMIT
• HOSTLIMIT_PER_JOB

The possible UNIX per-process resource usage limits are:

• RUNLIMIT
• FILELIMIT
• DATALIMIT
• STACKLIMIT
• CORELIMIT

If a job submitted to the queue has any of these limits specified (see bsub(1)), then the lower of the corresponding job limits and queue limits are used for the job.

If no resource limit is specified, the resource is assumed to be unlimited. User shell limits that are unlimited are not displayed.

EXCEPTION STATUS

Possible values for the exception status of a job include:

idle

The job is consuming less CPU time than expected. The job idle factor (CPU time/runtime) is less than the configured JOB_IDLE threshold for the queue and a job exception has been triggered.

overrun

The job is running longer than the number of minutes specified by the JOB_OVERRUN threshold for the queue and a job exception has been triggered.

underrun

The job finished sooner than the number of minutes specified by the JOB_UNDERRUN threshold for the queue and a job exception has been triggered.

Requested resources

Shows all the resource requirement strings you specified in the bsub command.

Execution rusage

This is shown if the combined RES_REQ has an rusage OR || construct. The chosen alternative will be denoted here.

Synchronous Execution

Job was submitted with the -K option. LSF submits the job and waits for the job to complete.

JOB_DESCRIPTION

The job description assigned by the user. This field is omitted if no job description has been assigned.

The displayed job description can contain up to 4094 characters.

MEMORY USAGE

Displays peak memory usage and average memory usage. For example:

MEMORY USAGE:

MAX MEM:11 Mbytes; AVG MEM:6 Mbytes

You can adjust rusage accordingly next time for the same job submission if consumed memory is larger or smaller than current rusage.

RESOURCE REQUIREMENT DETAILS

Displays the configured level of resource requirement details. The BJOBS_RES_REQ_DISPLAY parameter in lsb.params controls the level of detail that this column displays, which can be as follows:

• none - no resource requirements are displayed (this column is not displayed in the -l output).
• brief - displays the combined and effective resource requirements.
• full - displays the job, app, queue, combined and effective resource requirements.

Requested Network

Displays network resource information for IBM Parallel Edition (PE) jobs submitted with the bsub -network option. It does not display network resource information from the NETWORK_REQ parameter in lsb.queues or lsb.applications.

For example:

bjobs -l
Job <2106>, User <user1>;, Project <default>;, Status <RUN>;, Queue <normal>, Co
                     mmand <my_pe_job>
Fri Jun  1 20:44:42: Submitted from host <hostA>, CWD <$HOME>, Requested Network
                      <protocol=mpi: mode=US: type=sn_all: instance=1: usage=dedicated>

If mode=IP is specified for the PE job, instance is not displayed.

CLUSTER

The name of the cluster to which the job was forwarded.

FORWARD_TIME

The time that the job was forwarded.

JOBID

Job ID of the job array.

ARRAY_SPEC

Array specification in the format of name[index]. The array specification may be truncated, use -w option together with -A to show the full array specification.

OWNER

Owner of the job array.

NJOBS

Number of jobs in the job array.

PEND

Number of pending jobs of the job array.

RUN

Number of running jobs of the job array.

DONE

Number of successfully completed jobs of the job array.

EXIT

Number of unsuccessfully completed jobs of the job array.

SSUSP

Number of LSF system suspended jobs of the job array.

USUSP

Number of user suspended jobs of the job array.

PSUSP

Number of held jobs of the job array.

JOBID

Job ID of the Session Scheduler job.

OWNER

Owner of the Session Scheduler job.

JOB_NAME

The job name assigned by the user, or the command string assigned by default at job submission with bsub. If the job name is too long to fit in this field, then only the latter part of the job name is displayed.

The displayed job name or job command can contain up to 4094 characters for UNIX, or up to 255 characters for Windows.

NTASKS

The total number of tasks for this Session Scheduler job.

PEND

Number of pending tasks of the Session Scheduler job.

RUN

Number of running tasks of the Session Scheduler job.

DONE

Number of successfully completed tasks of the Session Scheduler job.

EXIT

Number of unsuccessfully completed tasks of the Session Scheduler job.

RUN

The job is running.

SSUSP

The job has been suspended by LSF.

USUSP

The job has been suspended, either by its owner or the LSF administrator, while running.

UNKNOWN

mbatchd has lost contact with the sbatchd on the host where the job was running.

PEND

The job is pending, which may include PSUSP and chunk job WAIT. When -sum is used with -p in MultiCluster, WAIT jobs are not counted as PEND or FWD_PEND. When -sum is used with -r, WAIT jobs are counted as PEND or FWD_PEND.

FWD_PEND

The job is pending and forwarded to a remote cluster. The job has not yet started in the remote cluster.

HOST

The host the task is running on

TYPE

Requested processor unit type for CPU binding. One of numa, socket, core, or thread.

LEVEL

Requested processor unit binding level for CPU binding. One of numa, socket, core, or thread. If no CPU binding level is requested, a dash (-) is displayed.

EXCL

Requested processor unit binding level for exclusive CPU binding. One of numa, socket, or core. If no exclusive binding level is requested, a dash (-) is displayed.

IDS

List of physical or logical IDs of the CPU allocation for the task.

The list consists of a set of paths, represented as a sequence integers separated by slash characters (/), through the topology tree of the host. Each path identifies a unique processing unit allocated to the task. For example, a string of the form 3/0/5/12 represents an allocation to thread 12 in core 5 of socket 0 in NUMA node 3. A string of the form 2/1/4represents an allocation to core 4 of socket 1 in NUMA node 2. The integers correspond to the node ID numbers displayed in the topology tree from bhosts -aff.

POL

Requested memory binding policy. Eitherlocal or pref. If no memory binding is requested, a dash (-) is displayed.

NUMA

ID of the NUMA node that the task memory is bound to. If no memory binding is requested, a dash (-) is displayed.

SIZE

Amount of memory allocated for the task on the NUMA node.

bsub -n 6 -R"span[hosts=1] rusage[mem=100]affinity[core(1,same=socket,
exclusive=(socket,injob)):cpubind=socket:membind=localonly:distribute=pack]" myjob
Job <6> is submitted to default queue <normal>.

bjobs -l -aff 6

Job <6>, User <user1>, Project <default>, Status <RUN>, Queue <normal>, Comman
                     d <myjob1>
Thu Feb 14 14:13:46: Submitted from host <hostA>, CWD <$HOME>, 6 Processors R
                     equested, Requested Resources <span[hosts=1] rusage[mem=10
                     0]affinity[core(1,same=socket,exclusive=(socket,injob)):cp
                     ubind=socket:membind=localonly:distribute=pack]>;
Thu Feb 14 14:15:07: Started on 6 Hosts/Processors <hostA> <hostA> <hostA
                     > <hostA> <hostA> <hostA>, Execution Home </home/user1
                     >, Execution CWD </home/user1>;

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -

 RESOURCE REQUIREMENT DETAILS:
 Combined: select[type == local] order[r15s:pg] rusage[mem=100.00] span[hosts=1
                     ] affinity[core(1,same=socket,exclusive=(socket,injob))*1:
                     cpubind=socket:membind=localonly:distribute=pack]
 Effective: select[type == local] order[r15s:pg] rusage[mem=100.00] span[hosts=
                     1] affinity[core(1,same=socket,exclusive=(socket,injob))*1
                     :cpubind=socket:membind=localonly:distribute=pack]

 AFFINITY:
                     CPU BINDING                          MEMORY BINDING
                     ------------------------             --------------------
 HOST                TYPE   LEVEL  EXCL   IDS             POL   NUMA SIZE
 hostA               core   socket socket /0/0/0          local 0    16.7MB
 hostA               core   socket socket /0/1/0          local 0    16.7MB
 hostA               core   socket socket /0/2/0          local 0    16.7MB
 hostA               core   socket socket /0/3/0          local 0    16.7MB
 hostA               core   socket socket /0/4/0          local 0    16.7MB
 hostA               core   socket socket /0/5/0          local 0    16.7MB