...
- Les turned on the debugging to investigate the discovery that several analysis scripts had stopped working in November 2019
- Each night we run about 100 analysis scripts as batch jobs and after fixing problems with the mon-lib.pl etc. we still found about 10% were randomly failing.
- After about 4 weeks it eventually transpired that some of the batch machines had not been enabled to acces access the Oracle database via the firewall. Thos The jobs submitted to those host failedhostsfailed.
- This The cause of the problem was eventually identified about January 3rd and fixed about January 6th. No data was lost.
Umar
State of MAs.
- The new version of Linux CentOS7, does not appear to support the following modules that are used by the PingER analysis.
- The Oracle database interface. PingER uses Oracle so there could be a problem.
- The perl statistics module
- I will restrict the PingER analysis jobs to the earlier version of Linux (RHEL6) to avoid these problems. I am working with the systems folks at SLAC and looking into when the current version of Linux at SLAC (RHEL6) will no longer be supported, and following SLAC's planning on how or whether to address the issues.
- The Oracle database interface. PingER uses Oracle so there could be a problem.
Umar
State of MAs.
Host | State | last seen | Status | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| pinger.nchc.org.tw | Intermittent problems starting 11/9/2019, it is not pingable starting 11/12 | |||||||||
Host | State | last seen | Status | |||||||
| pinger.nchc.org.tw | Intermittent problems starting 11/9/2019, it is not pingable starting 11/12/2019. | |||||||||
| pinger.vu.edu.pk | Unable to gather data since 29th August, 2019. The host is pingable. Emailed Wajahat 11/3/2019. Wajahat responded 11/11/2019 "Anything improved?". Les responded they are still not working with more details and suggesting we focus on vu. 11/20/2019 Les asks if there is an update? Wajanat responded 11/20/2019 "These guys are not responding. We are trying to contact them. Already tried multiple times." This just applies to vu. Wajahat recommends we keep trying to gather data. I have turned off the warnings. | 8/28/2019 | ||||||||
| monitor.seecs.edu.pk | No data since 10/29/2019, host is pingable email to Wajahat 11/7/2019, see vu for follow up. | 10/27/2019 | ||||||||
| maggie2.seecs.edu.pk | No data since 11/4/2019, host is pingable email to Wajahat 11/2/2019, 11/20/2019. See vu for follow up. | |||||||||
| pinger.nwfpuet.edu.pk | Something amiss since April 2019 when it dropped from monitoring over 150 targets down to 4. There is nothing in <BeaconsList>, all the hosts it attempts to monitor are in the locally manually maintained <NodeList>. Does not ping by name. Email sent 11/7/2019, and again 11/20/2019. | |||||||||
| pinger.uum.edu.my | Unable to gather data since 2nd May 2019. Host is pingable. Sent email 5/5/2019 to Adib, he will contact Dr. Suhaidi. Sent reminder 5/14/2019, Adib has contacted Dr Suhaidi who will get someone to help. Still down 6/2/2019, sent another reminder to Adib. Adib nudged Sohail. 6/19/2019 emailed Adib. 6/26/2019 Adib responded "Prof. Suhaidi has promised to follow up, but he was fully occupied / outstation early this month.Kindly give our friends some time ^_*". Sent reminder email 8/25/2019. Email 10/13/2019 to Adib and Sohail, will disable if no update by end October 2019. Suhaidi responded "Hi Les, will address this and get back to you soon. ". 11/3/2019 Still no data gathered, Disabled gathering and notified Sohail and Abid. | 5/2/2019 | Disabled | |||||||
| pinger.unimas.my | This has been unavailable for 6 months. Disabled as of end October since no response to emails. Notified Johari 11/3/2019. | 5/20/2019 | Disabled | |||||||
| pingeramity.in | Continues to be unstable. Email 8/29/2019. Turned off email warnings 11/20/2019. | pinger6 | ||||||||
| pinger. | csvu. | ubruedu. | ac.thpk | Unable to gather data since 29th August, 2019. The host is pingable. Emailed Wajahat 11/3/2019. Wajahat responded 11/11/2019 "Anything improved?". Les responded they are still not working with more details and suggesting we focus on vu. 11/20/2019 Les asks if there is an update? Wajanat responded 11/20/2019 "These guys are not responding. We are trying to contact them. Already tried multiple times." This just applies to vu. Wajahat recommends we keep trying to gather data. I have turned off the warnings. | 8/28/2019 | |||||
| monitor.seecs.edu.pk | No data since 10/29/2019, host is pingable email to Wajahat 11/7/2019, see vu for follow up. | 10/27/2019 | ||||||||
| maggie2.seecs.edu.pk | No data since 11/4/2019, host is pingable email to Wajahat 11/2/2019, 11/20/2019. See vu for follow up. | |||||||||
| pinger.nwfpuet.edu.pk | Something amiss since April 2019 when it dropped from monitoring over 150 targets down to 4. There is nothing in <BeaconsList>, all the hosts it attempts to monitor are in the locally manually maintained <NodeList>. Does not ping by name. Email sent 11/7/2019, and again 11/20/2019. | |||||||||
| pingeramity.in | Continues to be unstable. Email 8/29/2019. Turned off email warnings 11/20/2019. | Stopped being pingable 2/25/2019. Thus by default, the gathering failed. Emailed Charnsak 3/4/2019, 3/15/2019. Down again 4/23/2019, restored 4/27/2019. Down again 6/6/2019. It is pingable. Sent email 7/5/2019. No response. Sent another email 7/24/2019 it was not pingable. Fixed 8/15/2019, then down again 8/24/2019, it is pingable from SLAC. Unable to ping from http://pinger6.cs.ubru.ac.th/cgi-bin/traceroute.pl?function=ping&target=www.slac.stanford.edu&options=-i%200.2. Email 8/28/2019. Email 10/13/2019. Charnsak responded he plans to look at it. Charnsak responded 10/22/2019 "They have changed the Networking policy. The pinger may need to wait for 5-6 months for me to go back (from post-doc at University of Vienna) and change things myself." Host disabled 11/3/2019 | 9/29/2019 | Disabled | pinger.cs.ubru.ac.th | Down starting 8/19/2019, unable to ping from SLAC. Email 8/28/2019. Email 10/13/2019, Charnsak responded he plans to look at it. Charnsak responded he plans to look at it. Charnsak responded 10/22/2019 "They have changed the Networking policy. The pinger may need to wait for 5-6 months for me to go back (from post-doc at University of Vienna) and change things myself." Host disabled 11/3/2019 | 8/19/2019 | Disabled
Next Meeting
Next meeting: There will be a Doodle poll, for December or January 2020 at 10 pm Pacific time; a day later 10:00 am Pakistan time; 10:30 am India time; 1:00 pm Malaysian & Guangzhou time; 2:00 pm Thailand time; 7:00 am Jordan time. 6:00am Turkey time.
...