General information

Location of the scripts: /sdf/group/fermi/a/ground/dataQualityMonitoring/

Scripts to be tested: dev/bin/ingestTrendingFile and dev/bin/ingestRunFile

Temp area for testing: /sdf/group/fermi/ground/PipelineConfig/L1Proc/IngestTestArea

Testing environment

cd /sdf/group/fermi/ground/PipelineConfig/L1Proc/IngestTestArea
./start_rhel6	#load the container and its bind mounts
export JAVA_HOME=/sdf/group/fermi/software/java/jdk/jdk8.0
/sdf/group/fermi/a/ground/dataQualityMonitoring/dev/bin/ingestTrendingFile r0682587162_v001_digiTrend.root Prompt dev Digi 15secs

Debugging round 1

All ingestion jobs are failing with this error: ingestRunFile: line 2: /u/gl/glast/oracle/bin/ No such file or directory

Solution/workaround: all ingest files were upgraded by Max to use ~glast/a/oracle/bin/ at setup time.

Debugging round 2

The worker nodes don't seem to be resolving  ~glast/a/oracle/bin/ correctly (or possibly have access to it).

Solution/workaround (suggested by Richard): the ingest files in DEV were updated to source /sdf/home/glast/a/oracle/bin/

Debugging round 3🙈 

The pipeline env is setting  JAVA_HOME=/sdf/group/fermi/software/java/jdk/jdk8.0 but the correct path is /sdf/group/fermi/sw

Workaround (suggested by Richard): create a temporary symlink to /sdf/group/fermi/software for testing purposes only. Remember to remove!!!

Debugging round 4

The oracle setup script does:  export TNS_ADMIN=~glast/a/oracle/admin/ but the path won't expand correctly on worker nodes/containers

Solution (implemented by MEM): edit the setup script expanding the path: export TNS_ADMIN=/sdf/home/g/glast/a/oracle/admin/

  • No labels