You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

  • Specify an output file 

    Use the -o or -oo option to bsub to specify an output file for your batch job.  If you do not specify a viable file, the output will be sent via email which when multiplied by the 100s or 1000s can easily overwhelm the mail server.

  • Use local scratch space

    Use local scratch space during the course of your job and copy over the results to the final location when the job finishes.  Using local /scratch space is more efficient for constant writing and/or reading than nfs or AFS (i.e. over the network), and multiple batch jobs reading/writing to one nfs or AFS server can cause problems for the fileserver and everyone else who may be using it.

    The amount of scratch available varies by machine type, but you can request a host with a certain amount of space with the -R option, e.g.  bsub -R "scratch>10" would request a host with 10GB of scratch. A wrapper to submit your batch job is recommended which does the following:


      –   Create a directory in /scratch using the batch job ID ($LSB_JOBID).
      –   Copy any required input files to your /scratch directory.
      –   Write your program output to the newly created directory.
      –   When the program/script/command finishes copy the output file to a more permanent location.
      –   Remove your job directory from the scratch area.

     

  • Specify a wall clock time
    Use the -W or -We option to bsub to specify the length of time you expect your job to run.  This is preferable to specifying a queue because it allows the batch system to schedule your job more efficiently, taking advantage of backfill opportunities.

  • Specify the expected amount of memory needed
    Use the -R and -M options to bsub to specify the amount of memory that you expect your job to use.  For example,

       bsub -R "mem > 15000" -M 15000 <yourjob>

    The -R option tells bsub to schedule the job on a system with at least 15GB of memory.  The -M option tells LSF to allow your job to use up to 15GB of memory.  If your job exceeds that it will be killed.  Without this option, if your job exceeds the amount of memory available to the server, it will crash and take down all jobs currently running.  Since your job may be running along with jobs from other users, you would be hurting not only yourself, but others as well.
  • No labels