...

import os
import requests
requests.post(os.environ["JID_UPDATE_COUNTERS"], json=[ {"key": "<b>LoopCount</b>", "value": "75" } ])

2)

...

Examples.

The executable script used in the workflow definition should be used primarily to set up the environment etc and submit the analysis script to the HPC workload management infrastructure. For example, here's a simple executable script that uses LSF's bsub to submit the analysis script to the psdebugq queue

The following example scripts live at /reg/g/psdm/web/ws/test/apps/release/logbk_batch_client/test/submit.sh and /reg/g/psdm/web/ws/test/apps/release/logbk_batch_client/test/submit.py.

2.1) submit.sh

The script that the hash corresponds to is the one that submits the job via the bsub command. This script is shown below.

#!/bin/bash

ABS_PATH=/reg/gd/psdm/webdia/wsdiadaq13/test/apps/logbk_batch_client/testscratch/analyze
bsub -q psdebugq -o $ABS_PATH/logs/%J.log python $ABS_PATH/submit.py "$@"

This script will run the batch job submit /reg/d/psdm/dia/diadaq13/scratch/analyze on psdebugq and store the log files in /reg/gd/psdm/webdia/ws/test/apps/release/logbk_batch_client/testdiadaq13/scratch/logs/<lsf_id>. Also, it will pass all arguments passed to it to the python script, submit.py (these would be the parameters entered in the Batch defs tab)Since the job definition parameters and JID parameters are defined as environment variables, /reg/d/psdm/dia/diadaq13/scratch/analyze will inherit the EXPERIMENT, RUN_NUM and JID_UPDATE_COUNTERS environment variables.

2.2) submit.py

The Python script is the code that will do analysis and whatever is necessary on the run data. Since this is just an example, the Python script, submit.py, doesn't get that involved. It is shown below.

from time import sleep
from requests import post
from sys import argv
from os import environ
from numpy import random
from string import ascii_uppercase

print 'This is a test function for the batch submitting.\n'

## Fetch the URL to POST to
update_url = environ.get('BATCHJID_UPDATE_URLCOUNTERS')
print 'The update_url is:', update_url, '\n'

## Fetch the passed arguments as passed by submit.sh
params = argv
print 'The parameters passed are:'
for n, param in enumerate(params):
    print 'Param %d:' % (n + 1), param
print '\n'

## Run a loop, sleep a second, then POST
for i in range(10):
    sleep(1)
    rand_char = random.choice(list(ascii_uppercase))
 
    print 'Step: %d, %s' % (i + 1, rand_char)
    post(update_url, json={'counters' : [{'Example Counter' : [i + 1, 'red']},
                                          {'Random Char' : rand_char}}])

2.3) Log File Output

The print statements print out to the run's log file. The output of submit.py is below. The first parameter is the path to the Python script, the second is the experiment name, the third is the run number and the rest are the parameters passed to the script.

...

Can a submitted job submit other subjobs?

• Yes, in a standard LSF/SLURM fashion, BUT the ARP will not know about the subjobs.  Only jobs submitted through the ARP webpage are known to the ARP.

...

• The ARP keeps track of the hashtags for each run and the associated LSF/SLURM jobid.  That information allows the ARP to kill jobs.

...