...

Recently "psipmi power cycle" doesn't work on some nodes, with the ipmi interface become unpingable.  A workaround discovered by Ric Claus: bring up firefox on psnxserv, go to https://drp-srcf-cmp010-ipmi.pcdsn (for example), skip past the security warning messages in the browser, find login information in a file like /cds/group/pcds/admin/ipmicreds/drp-srcf-cmp010-ipmi.ini (these files are read-protected to members of the ps-ipmi group, you may have to "newgrp ps-ipmi").  Then there a menu in there to powercycle.  Takes longer, but Ric’s workaround is reliable in my experience.

NOTE: the web interface can report the operation as "failed" but it seems to work anyhow.  See bottom of this image:

Image Added

Dev Release Issue

On June 30th 2022 cpo notices that a new 4.5.16 release in TMO was picking up parts of the 4.5.13 release (which was creating an error).  The issue lasted about 18 hours until it was "fixed".  I could find no trace of 4.5.13 in any environment variable.  The same release 4.5.16 release worked in RIX.  In the end the problem was "fixed" by copying the tmo.cnf file to junk.cnf and editing the "cmd" field (in this case ami-client) to some other line.  After a start and stop of junk.cnf with this changed line tmo.cnf started working correctly.  Perhaps the remote procmgrd was caching the old 4.5.13 release since the "cmd" field hadn't changed?  Or, less likely, some the p0.cnf.last (or p0.cnf.running) was caching the old 4.5.13 release somehow?  I don't understand.  Or, said vaguely, there was some longterm nfs caching issue that remembered 4.5.13?

...