Content

Submit job in batch

bsub -o mylog0 -q psanaq ls -l
bsub -o mylog1 -q psanaq 'ls -l'
bsub -o mylog2 -q psanaq "ls -l"

bsub -o mylog -q psanaq curl -s "https://pswww-dev.slac.stanford.edu/calib_ws/"
[dubrovin@summitdev-login1:~]$ bsub -o mylog -q batch -W 1:00  -P CHM137 curl -s "https://pswww-dev.slac.stanford.edu/calib_ws/"

Useful system files

~/.ssh/known_hosts

Intercept batch node for interactive session

bsub -Is -q psnehprioq /bin/tcsh

Regular job submission to batch

bsub -q psfehq -o log-r169.log psana -c psana-config.cfg exp=cxif5315:run=169
bsub -q psfehq -o log-r169.log python psana-python.py

List of batch jobs

bjobs
bjobs -u dubrovin
bjobs -uall -q psfehhiprioq 

Status of hosts in the batch farm

List of LCLS farms: psnehfarm psfehfarm psanafarm

bhosts
bhosts psnehfarm psfehfarm psanafarm
bhosts | grep ps

Status of batch queues

list of LCLS queues:  psnehhiprioq psfehhiprioq psnehprioq psfehprioq psanaq psnehq psfehq

bqueues              - list all batch queuse
bqueues | grep ps
bqueues psnehhiprioq psfehhiprioq psnehprioq psfehprioq psanaq psnehq psfehq

List of batch hosts

lshost - listing of all batch machines along with their resources

lshosts | grep ps

Other commands

bqueues                     - Summary status of entire batch system organized by queues.
bqueues -l long - Detailed summary status of the 'long' queue.
lshosts         - Very long listing of all batch machines along with their resources.
bmod                            - Change the queue for a submitted job.
bhist                          - Get history information for completed jobs.
lsinfo                       - List all 'resources' defined in batch system.
busers                       - Summary of my batch activity.

qstat

 

Host specs

CPU

cat /proc/cpuinfo

Memory

free
cat /proc/meminfo

HDD

df -h
sudo fdisk -l
sudo hdparm -i /dev/<device> (where <device> for example sda1, hda3...)
sudo hdparm -i /dev/sda1
sudo dmidecode

 

Re-direct output in file

All POSIX operating systems have 3 streams: stdin, stdout, and stderr.

stdout is the primary output, which is redirected with >, >>, or |.

command >   file.txt
command >>  file.txt  # to append
command >&  file.txt # re-direct stdout and stderr
command 1>& file.txt # 1-stdout
command 2>& file.txt # 2-stderr
command &>> file.txt # append file with both streams (2-stderr to 1-stdout)
command 2> /dev/null # silence error
command 2>&1         # re-direct 2-stderr to 1-stdout
command 2>&1 >> file.txt # re-direct 2-stderr to 1-stdout and append file with both streams

 

Re-direct ls in file

USE BACKSLASH to get rid of formatting symbols in the output file

\ls -1 /reg/d/psdm/MEC > file.txt

Or similar with adding some content:

\ls -1 /reg/d/psdm/MEC | awk '{print "cdb convert -e " $1}' > test_convert_mec.sh

Save / re-direct text from command line in file

echo "abcd1234" > t.t

 

 

 

Make movie from images

convert _tmp*.png _tmp.gif

#or in python
os.system("convert _tmp*.png _tmp.gif")

 

Grab and save part of the screen in file

 

To grab portion of the screen imagemagic command import  can be used:

import -trim -frame -border file.jpeg

Works for *.jpeg, *.ppm

Resource limits can be seen by command

identify -list resource

 

Directory ACL

getfacl -h
getfacl /reg/g/psdm/logs/calibman/2016/

setfacl -h
sudo setfacl ...

User's group ids

id <username>

Restore deleted files

In home directory:

l .zfs/snapshot/

.zfs will be dynamically mounted.

Disk quota exceeded

If rm command does not work due to "Disk quota exceeded" try

cp /dev/null <your-big-file-name>

 

 

References

 

 

  • No labels