Chunk job dispatch

Jobs with the following characteristics are typical candidates for job chunking:
  • Take between 1 and 2 minutes to run

  • All require the same resource (for example a specific amount of memory)

  • Do not specify a beginning time (bsub -b) or termination time (bsub -t)

Running jobs with these characteristics without chunking can under utilize resources because LSF spends more time scheduling and dispatching the jobs than actually running them.

Configuring a special high-priority queue for short jobs is not desirable because users may be tempted to send all of their jobs to this queue, knowing that it has high priority.

Note:

Throughput can deteriorate if the chunk job size is too big. Performance may decrease on queues with CHUNK_JOB_SIZE greater than 30. You should evaluate the chunk job size on your own systems for best performance.

Restrictions on chunk jobs

CHUNK_JOB_SIZE is ignored and jobs are not chunked under the following conditions:
  • Interactive queues (INTERACTIVE = ONLY parameter)

  • CPU limit greater than 30 minutes (CPULIMIT parameter in lsb.queues or lsb.applications). If CHUNK_JOB_DURATION is set in lsb.params, the job is chunked only if it is submitted with a CPU limit that is less than or equal to the value of CHUNK_JOB_DURATION (bsub -c)

  • Run limit greater than 30 minutes (RUNLIMIT parameter in lsb.queues or lsb.applications). If CHUNK_JOB_DURATION is set in lsb.params, the job is chunked only if it is submitted with a run limit that is less than or equal to the value of CHUNK_JOB_DURATION (bsub -W)

  • Run time estimate greater than 30 minutes (RUNTIME parameter in lsb.applications)

Jobs submitted with the following bsub options are not chunked; they are dispatched individually:
  • -I (interactive jobs)

  • -c (jobs with CPU limit greater than 30)

  • -W (jobs with run limit greater than 30 minutes)

  • -app (jobs associated with an application profile that specifies a run time estimate or run time limit greater than 30 minutes, or a CPU limit greater than 30). CHUNK_JOB_SIZE is either not specified in the application, or CHUNK_JOB_SIZE=1, which disables chunk job dispatch configured in the queue.

  • -R "cu[]" (jobs with a compute unit resource requirement).