How do I keep programs running if a ssh connection fails

First, see if you can use the LSF batch nodes for your work. If not, three unix programs to help with this are tmux, nohup and screen. None of these programs will preserve a graphical program, or a X11 connection, so run your programs in terminal mode.

tmux

For example, with tmux, if one does

ssh psexport
ssh psana
# suppose we have landed on psanacs040 and that there is a matlab license here
tmux
matlab --nosplash --nodesktop 

If you lose the connection to psanacs040, you can go back to that node and reattach:

ssh psexport
ssh psanacs040
tmux attach

You do need to remember the node you ran tmux on.

nohup

You could run a batch process with nohup (no hangup) as follows

    nohup myprogram

For example, suppose we want to run a Python script that prints to the screen and save its output (the below syntax is for the bash shell):

nohup python myscript.py > myoutput 2>&1 &

Here we are capturing the output of the program in myoutput, along with anything it writes to stderr (the 2>&1), then putting it in the background. The job will persist after you logout. You can take a look at the output in the file myoutput the next day. You should remember the node you launched nohup on in case the job did not finish and you need to kill. If you ran Matlab in batch mode, you will lock up a license if it does not exit properly after running your script.

How do I run a LSF batch job?

Visit Submitting Batch Jobs, not this is for submitting batch jobs

Why did my batch job failed? I'm getting 'command not found'

...

The Psana ddl based Translator can be used to write ndarrays, strings and a few simple types that C++ modules register. These will be organized in the same groups that we use to translate xtc to hdf5. Datasets with event times will be written as well. To use this, create a psana config file that turns off the translation of all xtc types but allows translation of ndarrays and strings. An example cfg file is here: psana_translate_noxtc.cfg You would just change the modules and files parameters for psana and the output_file parameter to Translator.H5Output. Load modules before the translator that put ndarrays into the event store. The Translator will pick them up and write them to the hdf5 file

psexport01:~ $ ps -eu | grep davidsch
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
davidsch  9076  0.0  0.0   9544   860 pts/82   RN+  08:13   0:00 ps -eu HOSTNAME=psexport01 TERM=xterm SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=134.79.72.13 55848 22 MOZ_NO_REMOTE=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/82 SVN_EDITOR=emacs USER=davidsch LD_LIBRARY_PATH=./arch/x86_64-rhel5-gcc41-opt/lib:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/lib:/usr/lib64/openmpi/1.4-gcc/lib LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: SIT_DATA=./data:/reg/g/psdm/sw/releases/ana-current/data:/reg/g/psdm/data SIT_SVN=https://pswww.slac.stanford.edu/svn/psdmrepo SSH_AUTH_SOCK=/tmp/ssh-daFmw11286/agent.11286 KDEDIR=/usr SIT_ENV=ana SIT_ROOT=/reg/g/psdm SIT_RELDIR=/reg/g/psdm/sw/releases MAIL=/var/spool/mail/davidsch PATH=/reg/g/psdm/sw/external/scons/2.1.0-python2.7/bin:./arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/reg/neh/home/davidsch/local/bin:/reg/neh/home/davidsch/scripts LC_COLLATE=C INPUTRC=/etc/inputrc PWD=/reg/neh/home/davidsch SIT_REPOS=/reg/g/psdm/sw/releases/ana-current LANG=C SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass KRB5CCNAME=FILE:/tmp/krb5cc_12658_9oK8Nw SHLVL=1 HOME=/reg/neh/home/davidsch PYTHONPATH=./arch/x86_64-rhel5-gcc41-opt/python:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python:./arch/x86_64-rhel5-gcc41-opt/python/pyextra:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra LOGNAME=davidsch QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=134.79.72.13 55848 134.79.101.11 22 LESSOPEN=|/usr/bin/lesspipe.sh %s DISPLAY=localhost:52.0 SIT_SVN_USER=https://pswww.slac.stanford.edu/svn/userrepo SIT_RELEASE=ana-current G_BROKEN_FILENAMES=1 SIT_ARCH=x86_64-rhel5-gcc41-opt XAUTHORITY=/var/tmp/davidsch.Xauthority _=/bin/ps
davidsch  9077  0.0  0.0   6056   616 pts/82   RN+  08:13   0:00 grep davidsch HOSTNAME=psexport01 TERM=xterm SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=134.79.72.13 55848 22 MOZ_NO_REMOTE=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/82 SVN_EDITOR=emacs USER=davidsch LD_LIBRARY_PATH=./arch/x86_64-rhel5-gcc41-opt/lib:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/lib:/usr/lib64/openmpi/1.4-gcc/lib LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: SIT_DATA=./data:/reg/g/psdm/sw/releases/ana-current/data:/reg/g/psdm/data SIT_SVN=https://pswww.slac.stanford.edu/svn/psdmrepo SSH_AUTH_SOCK=/tmp/ssh-daFmw11286/agent.11286 KDEDIR=/usr SIT_ENV=ana SIT_ROOT=/reg/g/psdm SIT_RELDIR=/reg/g/psdm/sw/releases MAIL=/var/spool/mail/davidsch PATH=/reg/g/psdm/sw/external/scons/2.1.0-python2.7/bin:./arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/reg/neh/home/davidsch/local/bin:/reg/neh/home/davidsch/scripts LC_COLLATE=C INPUTRC=/etc/inputrc PWD=/reg/neh/home/davidsch SIT_REPOS=/reg/g/psdm/sw/releases/ana-current LANG=C SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass KRB5CCNAME=FILE:/tmp/krb5cc_12658_9oK8Nw SHLVL=1 HOME=/reg/neh/home/davidsch PYTHONPATH=./arch/x86_64-rhel5-gcc41-opt/python:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python:./arch/x86_64-rhel5-gcc41-opt/python/pyextra:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra LOGNAME=davidsch QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=134.79.72.13 55848 134.79.101.11 22 LESSOPEN=|/usr/bin/lesspipe.sh %s DISPLAY=localhost:52.0 SIT_SVN_USER=https://pswww.slac.stanford.edu/svn/userrepo SIT_RELEASE=ana-current G_BROKEN_FILENAMES=1 SIT_ARCH=x86_64-rhel5-gcc41-opt XAUTHORITY=/var/tmp/davidsch.Xauthority _=/bin/grep
davidsch 11287  0.0  0.0  67308  1716 pts/82   SNs  Feb03   0:00 -bash KRB5CCNAME=FILE:/tmp/krb5cc_12658_9oK8Nw USER=davidsch LOGNAME=davidsch HOME=/reg/neh/home/davidsch PATH=/usr/local/bin:/bin:/usr/bin MAIL= SHELL=/bin/bash SSH_CLIENT=134.79.72.13 55848 22 SSH_CONNECTION=134.79.72.13 55848 134.79.101.11 22 SSH_TTY=/dev/pts/82 TERM=xterm DISPLAY=localhost:52.0 XAUTHORITY=/var/tmp/davidsch.Xauthority SSH_AUTH_SOCK=/tmp/ssh-daFmw11286/agent.11286
davidsch 11635  0.0  0.0  67308  1704 pts/83   SNs+ Feb03   0:00 -bash KRB5CCNAME=FILE:/tmp/krb5cc_12658_7GPpft USER=davidsch LOGNAME=davidsch HOME=/reg/neh/home/davidsch PATH=/usr/local/bin:/bin:/usr/bin MAIL= SHELL=/bin/bash SSH_CLIENT=134.79.72.13 55849 22 SSH_CONNECTION=134.79.72.13 55849 134.79.101.11 22 SSH_TTY=/dev/pts/83 TERM=xterm DISPLAY=localhost:53.0 XAUTHORITY=/var/tmp/davidsch.Xauthority SSH_AUTH_SOCK=/tmp/ssh-WkRON11634/agent.11634
davidsch 11987  0.1  0.2 292968 55652 pts/83   SN   Feb03  13:14 /reg/neh/home/davidsch/emacs-24.3/bin/emacs-24.3 HOSTNAME=psexport01 TERM=xterm SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=134.79.72.13 55849 22 MOZ_NO_REMOTE=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/83 SVN_EDITOR=emacs USER=davidsch LD_LIBRARY_PATH=./arch/x86_64-rhel5-gcc41-opt/lib:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/lib:/usr/lib64/openmpi/1.4-gcc/lib LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: SIT_DATA=./data:/reg/g/psdm/sw/releases/ana-current/data:/reg/g/psdm/data SIT_SVN=https://pswww.slac.stanford.edu/svn/psdmrepo SSH_AUTH_SOCK=/tmp/ssh-WkRON11634/agent.11634 KDEDIR=/usr SIT_ENV=ana SIT_ROOT=/reg/g/psdm SIT_RELDIR=/reg/g/psdm/sw/releases MAIL=/var/spool/mail/davidsch PATH=/reg/g/psdm/sw/external/scons/2.1.0-python2.7/bin:./arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/reg/neh/home/davidsch/local/bin:/reg/neh/home/davidsch/scripts LC_COLLATE=C INPUTRC=/etc/inputrc PWD=/reg/neh/home/davidsch SIT_REPOS=/reg/g/psdm/sw/releases/ana-current LANG=C SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass KRB5CCNAME=FILE:/tmp/krb5cc_12658_7GPpft SHLVL=1 HOME=/reg/neh/home/davidsch PYTHONPATH=./arch/x86_64-rhel5-gcc41-opt/python:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python:./arch/x86_64-rhel5-gcc41-opt/python/pyextra:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra LOGNAME=davidsch QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=134.79.72.13 55849 134.79.101.11 22 LESSOPEN=|/usr/bin/lesspipe.sh %s DISPLAY=localhost:53.0 SIT_SVN_USER=https://pswww.slac.stanford.edu/svn/userrepo SIT_RELEASE=ana-current G_BROKEN_FILENAMES=1 SIT_ARCH=x86_64-rhel5-gcc41-opt XAUTHORITY=/var/tmp/davidsch.Xauthority _=/reg/neh/home/davidsch/emacs-24.3/bin/emacs-24.3
davidsch 11993  0.0  0.0  32116  4384 pts/83   SN   Feb03   0:00 /usr/libexec/gconfd-2 5 HOSTNAME=psexport01 TERM=xterm SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=134.79.72.13 55849 22 MOZ_NO_REMOTE=1 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/83 SVN_EDITOR=emacs USER=davidsch LD_LIBRARY_PATH=./arch/x86_64-rhel5-gcc41-opt/lib:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/lib:/usr/lib64/openmpi/1.4-gcc/lib LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: SIT_DATA=./data:/reg/g/psdm/sw/releases/ana-current/data:/reg/g/psdm/data SIT_SVN=https://pswww.slac.stanford.edu/svn/psdmrepo SSH_AUTH_SOCK=/tmp/ssh-WkRON11634/agent.11634 KDEDIR=/usr SIT_ENV=ana SIT_ROOT=/reg/g/psdm SIT_RELDIR=/reg/g/psdm/sw/releases MAIL=/var/spool/mail/davidsch PATH=/reg/g/psdm/sw/external/scons/2.1.0-python2.7/bin:./arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/bin:/reg/g/psdm/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/reg/neh/home/davidsch/local/bin:/reg/neh/home/davidsch/scripts LC_COLLATE=C INPUTRC=/etc/inputrc PWD=/reg/neh/home/davidsch SIT_REPOS=/reg/g/psdm/sw/releases/ana-current LANG=C SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass KRB5CCNAME=FILE:/tmp/krb5cc_12658_7GPpft SHLVL=1 HOME=/reg/neh/home/davidsch PYTHONPATH=./arch/x86_64-rhel5-gcc41-opt/python:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python:./arch/x86_64-rhel5-gcc41-opt/python/pyextra:/reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel5-gcc41-opt/python/pyextra LOGNAME=davidsch QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=134.79.72.13 55849 134.79.101.11 22 LESSOPEN=|/usr/bin/lesspipe.sh %s DISPLAY=localhost:53.0 SIT_SVN_USER=https://pswww.slac.stanford.edu/svn/userrepo SIT_RELEASE=ana-current G_BROKEN_FILENAMES=1 SIT_ARCH=x86_64-rhel5-gcc41-opt XAUTHORITY=/var/tmp/davidsch.Xauthority _=/reg/neh/home/davidsch/emacs-24.3/bin/emacs-24.3 GDK_CORE_DEVICE_EVENTS=1
psexport01:~ $