Page History

...

Red question marks show up in the display for HalfPipe. Click on the FASTCopy "Logs" number.  First check to see if the "ProcessSCI.end" event has occurred.  If not, then wait until it occurs and see if the red question mark persists.  If it persists and the log shows messages saying "pipeline nonEventReporting submission succeeded" or "no reassembled datagrams found", it just means that it contained data that had already been received. Meaningful deliveries will say "found XXX LSEP datagrams in ....".  If it is a meaningful delivery, then further intervention is required.

Stuck processes

If you suspect a process is stuck, visit the process page from the run page (ie - click on the green task bar in the Data Processing Page, then click on the particular job that is running in the list at the bottom of the page). Each entry in the table on the page will contain a link to "Messages", "Log", and "Files". If you click on "Log", it should show you the log, but more importantly, it'll show the location of the log file right above the box containing the log information (note: if the file is too big, it will not open, but the link to "Files" will show where the log is located, so you can get the information that way as well). If you visit the location of the file on the noric machines, you can determine when the last time the file was updated (ls -l, or tail -f to see if it's still streaming). If it is not updating, find out the batch job number (JobId from the stream table), log into noric as glastraw, and issue a bkill <JobId>. The process should automatically be restarted after this command is issued.

Batch commands

bjobs -wu glastraw - running this from noric will list all the jobs in the LSF batch queue owned by glastraw (which is the user submitting pipeline streams).

...