Timeline

I plan to release the new version to dev on March 19. I plan to release the new version to Prod on March 26.

Motivation:

1.4.1 Adds new support for cancelProcessInstance, parameterized jobsites in a batch process, and fixes bugs.

Notable new features:

1. 'Split mode' has been implemented as follows:

A process element in your XML definition can now contain a "site" attribute. This attribute will override the default site which is typically resolved through the task type. This site can be static, like SLAC, SLACMC, SLACRAW, or it can be a simple expression using a variable, or it can be an even more complex expression. Currently the expressions are limited to 64 characters, but we'll probably raise that if needed.

Examples include:

Static jobsite resolution:

 <process name="jobsiteProcess" site="SLAC">

Variable resolution [1]:

 <process name="jobsiteProcess" site="${jobsite}">

Complex conditional (Check to see if pipeline.stream is even, if it is, then SLAC, if it's not, then SLACMC)

 <process name="jobsiteProcess" site="${pipeline.stream %2 == 0 ? &quot;SLAC&quot; : &quot;SLACMC&quot;}">

[1] The variable jobsite can be declared with a default in the XML file, as you would with any variable, which can also be overridden at stream creation time.

2. cancelProcessInstance support in the linemode client.

This is a limited and fine grained tool for killing a specific submitted/running job. It is not a broad tool that cancelStream will need to be. I've been working on cancelStream but I couldn't get a robust, fast version out in time and tested for this release.

Example:

 pipeline cancelProcessInstance TaskName/SubTask processName 0.2
  •  where 0.2 is the streampath. The full taskpath, processname, and streampath are required. It can only kill one job at a time (but it's fast!) 
3. New Mbean for overriding the total number of jobs submitted and running in order to limit job submission. You can set this per site and/or per task.

A user can add an override in the jmx admin page. 

Where the old maxjobs is really a "max submitted", the MaxJobsTotal and MaxJobsTotalPerTask overrides will limit the maximum number of submitted AND running jobs.

For instance, you could limit MaxJobs to 200, and MaxJobsTotal to 1000. It would never allow more than 200 jobs in a submitted state, and it would never allow you to submit anymore if there were, say 100 submitted jobs and 900 running jobs. (for a total of 1000).

4. jython scriptlets can now use sys.exit() and exit with a code of 0 for success or any other code for failure.
5. Stream creation should be much quicker in most circumstances. 

Components to be upgraded:

Here is a list of the changes that will occur:

  • Stored procedures and Server move to version 1.4.1
  • Client:
    • org-srs-pipeline-client-1.1.0.jar -> org-srs-pipeline-client-1.1.1.jar

Related Jira Issues:

https://jira.slac.stanford.edu/browse/SSC-346

Issues relevant to this release:

https://jira.slac.stanford.edu/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+PII+AND+fixVersion+%3D+%221.4.1%22+AND+status+%3D+Resolved+ORDER+BY+priority+DESC&mode=hide1

Backward Compatibility:

Fully backward compatible

Upgrade Procedure:

Deploy new jars. Deploy stored procedures

Back-out Procedure:

Redeploy old jars. Redeploy old stored procedures

  • No labels