The batch processor allows for the easier submission of batch jobs via a web interface. A script that submits the batch job (to allow for more customization in this command) is all that is needed for this system to work.

1) Webpage

To use this system, choose an experiment from https://pswww-dev.slac.stanford.edu/apps-dev/portal/select_experiment.php.

1.1) Batch Job Definitions

Under the Experiment tab, the Batch defs tab is the location where scripts are stored. Below is an example where the example script is used which is described in detail below. A hash name is given to the absolute path of the script along with an arbitrary number of parameters which are separated by a space. An number of hashes can be created; as a hash is entered, a new row appears.

1.2) Batch Control

Over on Run Tables tab, the Batch control tab is where this hash may be applied to experiment runs. In the drop-down menu of the Action column, any hash defined in Batch defs may be selected. There are plans to add more options such as the ability to apply a hash to every run or to automatically apply a hash to a run in real-time.

Once a hash is applied to a run, it will appear as shown below. In this case, the example case has finished as shown by the DONE status (other statuses are described below). The last two columns warranted some explanation.

1.2.1) Actions

There are four different actions which can be applied to a script. They do the following if pressed:

 - Attempt to kill the job (via the bkill command). A green success statement will appear near the top-right of the page if the job is killed successfully and a red failure statement will appear if the job is not killed successfully.

 - Delete the hash from the run. Note: this does not kill the job, it only removes it from the webpage.

 - Returns the log file for the job. If there is no log file or if no log file could be found, it will return blank.

 - Returns details for the current job by invoking the "bjobs -l" command on the LSF ID.

1.2.2) Report

This is a customizable column which can be updated by the used script by posting to the correct URL. The URL is stored in the environment variable BATCH_UPDATE_URL. The counters shown in the screenshot above were done with the following syntax which was posted in a for loop in a python script: 

{'counters' : {'Example Counter' : [i + 1, 'red'], 'Random Char' : rand_char}}

As shown, the color of the output can also be customized. Whenever a POST is done for some submitted job (via the hash), the stored JSON for that job is updated only for what is posted. One value of this JSON is counters, along with others like 

Below is a simple example of a possible bash script. This script runs test.py and passes it any arguments that is passed to itself through "$@" and the log files are save in the logs directory with their job ID as the log file name. The python code can do analysis on the run this script was applied to

#!/bin/bash

bsub -q psdebugq -o LOG_DIRECTORY/%J.log python test.py "$@"

from time import sleep
from requests import post
from sys import argv
from os import environ
from numpy import random
from string import ascii_uppercase

print 'This is a test function for the batch submitting.\n'

update_url = environ.get('BATCH_UPDATE_URL')
print 'The update_url is:', update_url, '\n'

params = argv
print 'The parameters passed are:'
for n, param in enumerate(params):
    print 'Param %d:' % n, param
print '\n'

for i in range(10):
    sleep(1)
    rand_char = random.choice(list(ascii_uppercase))
 
    print 'Step: %d, %s' % (i + 1, rand_char)
    post(update_url, json={'counters' : {'Example Counter' : [i + 1, 'red'],
                                         'Random Char' : rand_char}})