Page History

...

On June 30th 2022 cpo notices that a new 4.5.16 release in TMO was picking up parts of the 4.5.13 release (which was creating an error).  The issue lasted about 18 hours until it was "fixed".  I could find no trace of 4.5.13 in any environment variable.  The same release 4.5.16 release worked in RIX.  In the end the problem was "fixed" by copying the tmo.cnf file to junk.cnf and editing the "cmd" field (in this case ami-client) to some other line.  After a start and stop of junk.cnf with this changed line tmo.cnf started working correctly.  Perhaps the remote procmgrd was caching the old 4.5.13 release since the "cmd" field hadn't changed?  Or, less likely, some the p0.cnf.last (or p0.cnf.running) was caching the old 4.5.13 release somehow?  I don't understand.  Or, said vaguely, there was some longterm nfs caching issue that remembered 4.5.13?

Ric's

...

Idea

I was having this problem as well.  I think it may have something to do with dmypy processes that hang around.  If one can get this guy to restart, the problem goes away.  For me, there were a bunch:

(ps-4.5.16) claus@drp-neh-ctl001:srcf$ ps -ef | grep dmypy
claus     61073      1  0 Jan07 ?        00:00:05 /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.5/bin/python /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.5/bin dmypy start
claus     61608      1  0 Apr01 ?        00:00:04 /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.11/bin/python /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.11/bin/dmypy start
claus    285627 222348  0 19:32 pts/9    00:00:00 grep dmypy
claus    353768      1  0 Jun09 ?        00:00:59 /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.13/bin/python /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.13/bin/dmypy start
claus    360001      1  0 Jun09 ?        00:01:02 /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.13/bin/python /cds/sw/ds/ana/conda2/inst/envs/ps-4.5.13/bin/dmypy start

I tried running ami-client on a different machine, which didn’t have an issue.  It also printed 'Daemon started’ whereas on the usual machine I run it on it printed 'Daemon is still alive’.  When I switched back to running on the usual machine, a new daemon was started and ami-client ran cleanly.

I found on https://mypy.readthedocs.io/en/stable/mypy_daemon.html that there is a ‘dmypy stop’ command, but it doesn’t seem to stop the existing daemons, so I killed ‘em.