I suggest the following processing states:

  • WAITING - waiting on some condition
  • READY - conditions are fulfilled, waiting to be queued
  • QUEUED - queued for execution
  • SUBMITTED - (batch jobs only)
  • RUNNING
  • FAILED
  • DONE

When a new stream is created we can create ProcessInstance entries for each process, with the initial ProcessState set to either READY or WAITING depending on whether it has any dependencies, and whether they are met.

The main loop can then simply search for processes in state "READY" and if so move them to the execution queue.

When a job completes it should search for any ProcessInstances that are waiting on it, and set the state to READY. This would remove the need for a complicated query in the main loop.

  • No labels