General information
Location of the scripts: /sdf/group/fermi/a/ground/dataQualityMonitoring/
Scripts to be tested: dev/bin/ingestTrendingFile and dev/bin/ingestRunFile
Temp area for testing: /sdf/group/fermi/ground/PipelineConfig/L1Proc/IngestTestArea
Testing environment
cd /sdf/group/fermi/ground/PipelineConfig/L1Proc/IngestTestArea ./start_rhel6 #load the container and its bind mounts export JAVA_HOME=/sdf/group/fermi/software/java/jdk/jdk8.0 /sdf/group/fermi/a/ground/dataQualityMonitoring/dev/bin/ingestTrendingFile r0682587162_v001_digiTrend.root Prompt dev Digi 15secs
Debugging round 1 ✅
All ingestion jobs are failing with this error: ingestRunFile: line 2: /u/gl/glast/oracle/bin/setup.sh: No such file or directory
Solution/workaround: all ingest files were upgraded by Max to use ~glast/a/oracle/bin/setup.sh at setup time.
Debugging round 2 ✅
The worker nodes don't seem to be resolving ~glast/a/oracle/bin/setup.sh correctly (or possibly have access to it).
Solution/workaround (suggested by Richard): the ingest files in DEV were updated to source /sdf/home/glast/a/oracle/bin/setup.sh
Debugging round 3🙈
The pipeline env is setting JAVA_HOME=/sdf/group/fermi/software/java/jdk/jdk8.0 but the correct path is /sdf/group/fermi/sw
Workaround (suggested by Richard): create a temporary symlink to /sdf/group/fermi/software for testing purposes only. Remember to remove!!!
Debugging round 4✅
The oracle setup script does: export TNS_ADMIN=~glast/a/oracle/admin/ but the path won't expand correctly on worker nodes/containers
Solution (implemented by MEM): edit the setup script expanding the path: export TNS_ADMIN=/sdf/home/g/glast/a/oracle/admin/