You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 43 Next »

Ric Issues

  • PCIe issues in which various cards don't come up right, I think we’ve seen this with Enet, IB and KCUs now
  • ATCA cards not powering up correctly; maybe a sequencing issue?
  • The incredibly long boot times of some nodes sometimes

HSD

UNSOLVED: early procmgr telnet timeout?

(ps-3.1.11) daq-tst-dev03:cnf$ procmgr start hsd.cnf
/reg/neh/home/cpo/git/lcls2/install/bin/procmgr: using config file 'hsd.cnf' to start
*** ERR: no prompt at drp-tst-acc06 port 29400
*** ERR: no prompt at drp-tst-dev016 port 29400
*** ERR: no prompt at drp-tst-dev020 port 29400
*** ERR: no prompt at drp-tst-dev021 port 29400
*** ERR: no prompt at drp-tst-dev022 port 29400
(ps-3.1.11) daq-tst-dev03:cnf$ telnet drp-tst-acc06 29400
Trying 172.21.148.224...
Connected to drp-tst-acc06.
Escape character is '^]'.
@@@ Welcome to procServ (procServ Process Server 2.6.0-SLAC)
@@@ Use ^X to kill the child, auto restart is ON, use ^T to toggle auto restart
@@@ procServ server PID: 174485
@@@ Server startup directory: /
@@@ Child startup directory: /tmp
@@@ Child started as: /bin/tcsh
@@@ Child "/bin/tcsh" PID: 174541
@@@ procServ server started at: Thu Jun  4 17:23:15 2020
@@@ Child "/bin/tcsh" started at: Thu Jun  4 17:23:15 2020
@@@ 0 user(s) and 0 logger(s) connected (plus you)
> 
telnet> q
Connection closed.
(ps-3.1.11) daq-tst-dev03:cnf$ date
Thu Jun  4 17:41:18 PDT 2020
(ps-3.1.11) daq-tst-dev03:cnf$



SOLVED: hsdpva crash on startup.  p4p get timeout too short? default is 5, try increase to 10.  Real solution: make hsdpvs (hsd pv server) long-lived procmgr process.

(ps-3.1.11) daq-tst-dev03:cnf$ cat /reg/neh/home/cpo/2020/06/04_17:39:45_daq-tst-dev03.pcdsn:hsdpva.log
# ID:      hsdpva
# PLATFORM:4
# HOST:    daq-tst-dev03.pcdsn
# CMDLINE: hsdpva DAQ:LAB2:HSD:DEV07_1A:A DAQ:LAB2:HSD:DEV07_1A:B DAQ:LAB2:HSD:DEV07_88:A DAQ:LAB2:HSD:DEV07_88:B DAQ:LAB2:HSD:DEV07_89:A DAQ:LAB2:HSD:DEV07_89:B DAQ:LAB2:HSD:DEV07_B1:A DAQ:LAB2:HSD:DEV07_B1:B DAQ:LAB2:HSD:DEV07_B2:A DAQ:LAB2:HSD:DEV07_B2:B
# TESTRELDIR:/reg/neh/home/cpo/git/lcls2/install
# CONDA_REL:ps-3.1.11
# GIT_DESCRIBE:3.0.4-39-ga30d0131
@@@ Restarting child "2020/06/04_17:39:45_daq-tst-dev03.pcdsn:hsdpva.log"
@@@    (as /bin/env)
@@@ The PID of new child "2020/06/04_17:39:45_daq-tst-dev03.pcdsn:hsdpva.log" is: 23772
@@@ @@@ @@@ @@@ @@@
5.12.3
Traceback (most recent call last):
  File "/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/lib/python3.7/site-packages/p4p/client/thread.py", line 270, in get
    value, i = done.get(timeout=timeout)
  File "/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/lib/python3.7/queue.py", line 178, in get
    raise Empty
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/reg/neh/home/cpo/git/lcls2/install/bin/hsdpva", line 11, in <module>
    load_entry_point('psdaq', 'console_scripts', 'hsdpva')()
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/cas/hsdpva.py", line 349, in main
    ui.setupUi(MainWindow,args.base)
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/cas/hsdpva.py", line 313, in setupUi
    maintab.addTab( HsdJesd         (title), 
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/cas/hsdpva.py", line 245, in __init__
    pvTtl.get()
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/cas/pvedit.py", line 78, in get
    self.__value__ = pvactx.get(self.pvname)
  File "/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/lib/python3.7/site-packages/p4p/client/thread.py", line 274, in get
    raise TimeoutError()
TimeoutError



SOLVED: See 100% dead time from all 5 hsd's.  Ric suggests looking at the Read Buffers section in /proc/datadev_0 (and 1).  this suggests that all buffers are in the kcu:  (not stuck in software).  disable times out.  answer (after working with Matt) msgdelay was set to 91 in hsdpva: too short so overwrote front-end buffers using "pvput DAQ:LAB2:XPM:2:PART:4:L0Delay 98".  setting it to 98 displays it as 99 for reasons I don't understand.

  Buffers In User : 0          (number of buffers in software)
  Buffers In Hw : 4095         (number of buffers in the kcu)
  Buffers In Pre-Hw Q : 61334  (number of buffers driver has available to hand to the kcu)
  Buffers In Rx Queue : 0      (buffers transferred from kcu to driver, ready to be received by dmaReadBulkIndex)



UNSOLVED: groupca crashes with a p4p timeout.  trace it back to pva get() working but put() failing:

(ps-3.1.11) daq-tst-dev03:cnf$ pvget DAQ:LAB2:XPM:2:PART:4:Master
DAQ:LAB2:XPM:2:PART:4:Master 2020-06-05 17:09:25.695  1 
(ps-3.1.11) daq-tst-dev03:cnf$ pvput DAQ:LAB2:XPM:2:PART:4:Master 1
Old : 2020-06-05 17:09:25.695  1 
Put timeout
(ps-3.1.11)

in turn trace it down to a crash in pyxpm-2:

Group: 4  Master: 1  RateSel: 0  DestSel: 8000  Ena: 1
Group: 3  Master: 1  RateSel: 3  DestSel: 8000  Ena: 0
Group: 7  Master: 1  RateSel: 4  DestSel: 8000  Ena: 0
Traceback (most recent call last):
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/pyxpm/pvstats.py", line 457, in update
    self._links[i].update()
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/pyxpm/pvstats.py", line 136, in update
    updatePv(self._pv_remoteLinkId,self._app.remId.get())
  File "/reg/neh/home4/cpo/git/lcls2/psdaq/psdaq/pyxpm/pvstats.py", line 114, in updatePv
    pv.post(value)
  File "/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/lib/python3.7/site-packages/p4p/server/raw.py", line 160, in post
    _SharedPV.post(self, self._wrap(value))
RuntimeError: bad_weak_ptr
Caught exception... retrying.

Sent email to Michael Davidsaver, Hugo Slepicka and the control group to see if they had seen this before.  They had not.  Michael wrote:  "At a guess, a
client disconnect will be involved."

SOLVED (1MHz): running 5 hsd nodes at 1mhz saw this on dev010 and node became unresponsive (disable timed out?).  removed dev010 but saw disable timeout on dev019:

Message from syslogd@drp-tst-dev010 at Jun  5 13:02:43 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [swapper/10:0]

problem goes away if we reduce the trigger rate to 71kHz.  even "ls" on the drp node will hang until the trigger rate is reduced to 71kHz.  "ls" hanging is reproducible even when running only 1 of the two hsd's on a drp.  pgp driver and firmware haven't changed since February.  caused by conda compiler work? no. can see problem with pgpread with both old/new compilers.  maybe it's having 2 hsd's on a drp (interrupts may still be firing even when we don't run the second hsd drp executable).  Matt has been unable to reproduce with fakecam so far.  Update: found that even running 1 hsd that interrupts for both hsd's were firing (saw counters in /proc/datadev_1 incrementing).  Followed Matt's advice and "unconfigure" disabled the second hsd's interrupts, and in this configuration there were no more soft lockups.  Checked that it behaved with datadev_0 or datadev_1 running individually, just not both together.  Solution: add an interrupt-hold-off parameter to the firmware to throttle maximum interrupt rate (in addition to the existing pattern of only-interrupt-when-queue-goes-non-empty).


UNSOLVED (1MHz): with both the fake cam and hsd Matt saw that if he ran at 1MHz then disabled for a few seconds then reenabled that all the buffers were stuck in software and system would hang.  Reproducible 1 out of 3 attempts.


UNSOLVED (1MHz): when we record 10 hsd's to disk disable times out for several segment levels.  No errors in logfiles.  Timeouts too short for 1MHz?


UNSOLVED: hsd configure times out the first time, but works second time.  Matt says that he seen the transition not received at all by the hsd.  Phase 1 completes in Digitizer.cc but phase2 maybe doesn't start, consistent with Matt's observation?  log file in /reg/neh/home/cpo/2020/06/12_17:41:55_drp-tst-dev010:tmohsd_0.log.  What I observed: both tmohsd_0/1 completed phase 1 but failed to receive phase 2.  How I think it's supposed to work:  tmohsd_0/1 each write some kcu registers (resetting tx/rx/user/qpll).  both tmohsd_0/1 set config epics vars in its own pva server ("hsdpvs" process).  both of these servers communicate with one "hsdioc" (running hsd134PVs c++ executable on dev07).  hsdioc programs both channels of the hsd ("fmc" in the code) and sets READY (e.g. DAQ:LAB2:HSD:DEV07_1A:B:READY).  tmohsd_0/1 both watch for their own A:READY and B:READY for each channel.  maybe resets take a while to recover?  maybe a "reentrancy" problem in hsdioc?Could it be the clear readout?  But we have a 1s delay in control.py after clear readout (sent on both configure/beginrun).

In these log files saw that only one half of the hsd didn't get phase2: /reg/neh/home/cpo/2020/06/15_18:39:03_drp-tst-dev021:tmohsd_7.log /reg/neh/home/cpo/2020/06/15_18:39:03_drp-tst-dev021:tmohsd_6.log.


UNSOLVED: according to hsdpva only one hsd was producing dead time (due to msgdelay being set too short) but xpmpva showed all hsd's having dead time.  at other times xpmpva seems to be attributing dead time correctly, so it's somehow an intermittent issue, perhaps only happening when Ben detects his front-end buffers have overflowed?  (which causes him to latch dead time until receiving clear readout).

 

PROPOSED SOLUTION: running 1 hsd (datadev_0) and after a while the deadtime goes to 100% from that hsd, but timepausecnt is zero for both hsd’s.  I think it's caused by ben's headercntof latching to 1 on the “A” hsd, even though msgdelay is set to 99.   Do we need to increase msgdelay?  Matt says msgdelay is max 100 (corresponding to 1us).  This could be caused by slow updates ignoring dead time if we get long dead time due to the soft-lockup issue above (now solved by Matt, we believe).  So could be the same issue.  Matt will have SlowUpdates pay attention to dead time to avoid this.


UNSOLVED: got a segv in this log file: /reg/neh/home/cpo/2020/06/15_18:39:03_daq-tst-dev03.pcdsn:hsdpva.log.  restarting the process fixed it.  Later, generated a core corresponding to this log /reg/neh/home/cpo/2020/06/16_15:49:13_daq-tst-dev03.pcdsn:hsdpva.log:

gdb `which python` ./core.hsdpva.13782
 
(gdb) where
#0  0x00007ff0942ed1f7 in raise () from /lib64/libc.so.6
#1  0x00007ff0942ee8e8 in abort () from /lib64/libc.so.6
#2  0x00005631b9a7ced3 in fatal_error (prefix=prefix@entry=0x0, 
    msg=msg@entry=0x5631b9c4f5b9 "GC object already tracked", 
    status=status@entry=-1)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pylifecycle.c:2187
#3  0x00005631b9a7cf34 in Py_FatalError (
    msg=msg@entry=0x5631b9c4f5b9 "GC object already tracked")
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pylifecycle.c:2197
#4  0x00005631b9a84b0b in PyTuple_New (size=size@entry=2)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/tupleobject.c:133
#5  0x00005631b9b3bb85 in _PyStack_AsTuple (stack=stack@entry=0x5631bbaecb00, 
    nargs=nargs@entry=2)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:1284
#6  0x00005631b9b82149 in _PyObject_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:177
#7  0x00005631b9be7a16 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
---Type <return> to continue, or q <return> to quit---
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#8  _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#9  0x00005631b9b2bbda in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#10 0x00005631b9b7a9e7 in _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
#11 0x00005631b9be34a0 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#12 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3110
#13 0x00005631b9b2c0b8 in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#14 0x00005631b9b2cc60 in _PyFunction_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Obje---Type <return> to continue, or q <return> to quit---
cts/call.c:376
#15 0x00005631b9b2cdd6 in _PyObject_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
#16 0x00005631b9b4ae03 in _PyObject_Call_Prepend ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
#17 0x00005631b9b815fa in slot_tp_init ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#18 0x00005631b9b82208 in type_call (kwds=0x7ff04c08cf00, args=0x7ff0550b0c30, 
    type=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#19 _PyObject_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#20 0x00005631b9be85c6 in call_function (kwnames=0x7ff0550ef250, 
    oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#21 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/ceval.c:3139
#22 0x00005631b9b2b8f9 in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#23 0x00005631b9b7a9e7 in _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
#24 0x00005631b9be42e7 in call_function (kwnames=0x7ff08cb636d0, 
    oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#25 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3139
#26 0x00005631b9b2bbda in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#27 0x00005631b9b2ca35 in _PyFunction_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:376
#28 0x00005631b9b2cdd6 in _PyObject_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
---Type <return> to continue, or q <return> to quit---
#29 0x00005631b9b4ae03 in _PyObject_Call_Prepend ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
#30 0x00005631b9b815fa in slot_tp_init ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#31 0x00005631b9b82208 in type_call (kwds=0x0, args=0x7ff05507cdc0, 
    type=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#32 _PyObject_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#33 0x00005631b9be7a16 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#34 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#35 0x00005631b9b7a75b in function_code_fastcall (globals=<optimized out>, 
    nargs=3, args=<optimized out>, co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Obje---Type <return> to continue, or q <return> to quit---
cts/call.c:283
#36 _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#37 0x00005631b9be34a0 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#38 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3110
#39 0x00005631b9b7a75b in function_code_fastcall (globals=<optimized out>, 
    nargs=0, args=<optimized out>, co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:283
#40 _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#41 0x00005631b9be3216 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#42 _PyEval_EvalFrameDefault ()
---Type <return> to continue, or q <return> to quit---
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#43 0x00005631b9b2b8f9 in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#44 0x00005631b9b2c824 in PyEval_EvalCodeEx ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3959
#45 0x00005631b9b2c84c in PyEval_EvalCode (co=co@entry=0x7ff0949dd8a0, 
    globals=globals@entry=0x7ff094a131e0, locals=locals@entry=0x7ff094a131e0)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:524
#46 0x00005631b9c42684 in run_mod ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:1035
#47 0x00005631b9c4cab1 in PyRun_FileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:988
#48 0x00005631b9c4cca3 in PyRun_SimpleFileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:429
#49 0x00005631b9c4cd6d in PyRun_AnyFileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/pythonrun.c:84
#50 0x00005631b9c4dd95 in pymain_run_file (p_cf=0x7fff187f5c40, 
    filename=0x5631bb10e980 L"/reg/neh/home/cpo/git/lcls2/install/bin/hsdpva", 
    fp=0x5631bb16a8c0)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:428
#51 pymain_run_filename (cf=0x7fff187f5c40, pymain=0x7fff187f5d50)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:1607
#52 pymain_run_python (pymain=0x7fff187f5d50)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:2868
#53 pymain_main ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3029
#54 0x00005631b9c4debc in _Py_UnixMain ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3064
#55 0x00007ff0942d9c05 in __libc_start_main () from /lib64/libc.so.6
#56 0x00005631b9bf2bd0 in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb)



UNSOLVED: Saw this error in /reg/neh/home/cpo/2020/06/15_14:39:26_daq-tst-dev07:hsdioc_89.log when starting up processes after a power-cycle (for new hsd firmware for the interrupt-holdoff):

ADC Initial Lane Alignment Failed!
reg7 = 0xC

 hsd "89" was on the slave XPM:1.  another hsd "88" was on the same XPM:1 but worked.  all links looked locked on the xpm side.   The various digitizers look similar:

Restarting the hsdioc_89 process "fixed" the problem.

 

UNSOLVED: Running at 10Hz with no recording phase2 of disable not be seen by any hsds: /reg/neh/home/cpo/2020/06/16_15:26:24_drp-tst-acc06:control.log.  can see control.py try to write xpm pv's for disable phase 2.

/reg/neh/home/cpo/2020/06/16_15:26:24_drp-tst-acc06:control.log

/reg/neh/home/cpo/2020/06/16_15:26:24_drp-tst-acc06:control.log

(ps-3.1.11) daq-tst-dev03:lcls2$ gdb `which python` ./psdaq/psdaq/cnf/core.hsdpva.23738
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/bin/python3.7...done.

warning: core file may not match specified executable file.
[New LWP 23738]
[New LWP 24583]
[New LWP 24585]
[New LWP 24580]
[New LWP 24589]
[New LWP 24579]
[New LWP 24630]
[New LWP 24582]
[New LWP 24581]
[New LWP 24584]
[New LWP 24615]
[New LWP 25101]
[New LWP 24619]
[New LWP 25163]
[New LWP 25156]
[New LWP 25099]
[New LWP 24617]
[New LWP 25716]
[New LWP 25441]
[New LWP 25758]
[New LWP 25160]
[New LWP 25783]
[New LWP 25771]
[New LWP 25784]
[New LWP 25803]
[New LWP 25798]
[New LWP 25757]
[New LWP 25772]
[New LWP 25155]
[New LWP 25752]
[New LWP 25715]
[New LWP 25801]
[New LWP 25799]
[New LWP 25440]
[New LWP 25751]
[New LWP 25787]
[New LWP 25782]
[New LWP 24629]
[New LWP 24634]
Missing separate debuginfo for /lib64/libpthread.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/60/91139b61d901aa8425dbc16b7f51d455a416b4.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /lib64/libc.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ff/1d2de4cf353118960c5ea195ceb13699929c14.debug
Missing separate debuginfo for /lib64/libdl.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ef/023e0c16991e8957337c90b0e51025fe27c897.debug
Missing separate debuginfo for /lib64/libutil.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b8/082f4480af5af680a0ad0ab4818cae1c077a0b.debug
Missing separate debuginfo for /lib64/librt.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/76/c4765826961c60be913f5549ebbc7bbe506899.debug
Missing separate debuginfo for /lib64/libm.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/6a/344a5ce77a247399aa2f22f92c698f574a7134.debug
Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a9/980cf253c79740e69f70dcb8fea7b8c2f641b5.debug
Missing separate debuginfo for /lib64/libGL.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a2/f537e16380004afe5920223ba4837a7cf9193f.debug
Missing separate debuginfo for /lib64/libxshmfence.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/09/4eafa9d4305b9d112a97d33a68361fe72073b7.debug
Missing separate debuginfo for /lib64/libglapi.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/74/370865c18663a14d50292b39ba4589a049e4e7.debug
Missing separate debuginfo for /lib64/libselinux.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/6a/5ef7a05f7e488fcd280bfadd96083bec9fd416.debug
Missing separate debuginfo for /lib64/libXdamage.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f2/709bb30b8496f436775e13d2250d5565fa8b64.debug
Missing separate debuginfo for /lib64/libXfixes.so.3
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/062d2deb4d533956a51bcf82f487e2bcd14c11.debug
Missing separate debuginfo for /lib64/libXxf86vm.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/56/9cf7657010c08e28262476726d80eb9bb8ea02.debug
Missing separate debuginfo for /lib64/libdrm.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/37/7b9e3631cf45fa052067dce55171b3c0340bfa.debug
Missing separate debuginfo for /lib64/libXcursor.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/27/ce4043698294568faa1373d44a82f74e49f08b.debug
Core was generated by `/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/bin/python /reg/neh/home/cpo/git/lcls'.
Program terminated with signal 11, Segmentation fault.
#0 0x00005618d53cb360 in ?? ()
(gdb) where
#0 0x00005618d53cb360 in ?? ()
#1 0x00005618d2da2c5f in visit_decref (op=<optimized out>,
data=<optimized out>, op=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:306
#2 0x00005618d2e3bde7 in list_traverse (o=0x7f31f805ad70,
visit=0x5618d2e3fc40 <visit_decref>, arg=0x0)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/listobject.c:2583
#3 0x00005618d2e68f74 in subtract_refs ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:296
#4 collect ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:853
#5 0x00005618d2e69a24 in collect_with_callback ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1025
#6 0x00005618d2f2f939 in collect_generations ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1048
#7 0x00005618d2e69b84 in _PyObject_GC_Alloc (basicsize=<optimized out>,
use_calloc=0)
---Type <return> to continue, or q <return> to quit---
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1705
#8 _PyObject_GC_Malloc ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1715
#9 0x00005618d2e69bae in _PyObject_GC_New (
tp=tp@entry=0x5618d3083ce0 <PyMethod_Type>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1727
#10 0x00005618d2e6dd92 in PyMethod_New ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/classobject.c:62
#11 0x00005618d2ea6cf0 in super_getattro ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:7570
#12 0x00005618d2f08c0c in _PyObject_GetMethod ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/object.c:1137
#13 0x00005618d2f1238e in _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3040
#14 0x00005618d2e5abda in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/ceval.c:3930
#15 0x00005618d2ea99e7 in _PyFunction_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
#16 0x00005618d2f124a0 in call_function (kwnames=0x0, oparg=<optimized out>,
pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#17 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3110
#18 0x00005618d2e5b0b8 in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#19 0x00005618d2e5bc60 in _PyFunction_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:376
#20 0x00005618d2e5bdd6 in _PyObject_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
#21 0x00005618d2e79e03 in _PyObject_Call_Prepend ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
---Type <return> to continue, or q <return> to quit---
#22 0x00005618d2eb05fa in slot_tp_init ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#23 0x00005618d2eb1208 in type_call (kwds=0x7f31bc7ba820, args=0x7f32024c1b90,
type=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#24 _PyObject_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#25 0x00005618d2f175c6 in call_function (kwnames=0x7f32024fe250,
oparg=<optimized out>, pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#26 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3139
#27 0x00005618d2e5a8f9 in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#28 0x00005618d2ea9985 in _PyFunction_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
---Type <return> to continue, or q <return> to quit---
#29 0x00005618d2f12216 in call_function (kwnames=0x0, oparg=<optimized out>,
pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#30 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#31 0x00005618d2e5abda in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#32 0x00005618d2e5bc60 in _PyFunction_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:376
#33 0x00005618d2e5bdd6 in _PyObject_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
#34 0x00005618d2e79e03 in _PyObject_Call_Prepend ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
#35 0x00005618d2eb05fa in slot_tp_init ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#36 0x00005618d2eb1208 in type_call (kwds=0x7f31bc7ad230, args=0x7f320248cc30,
---Type <return> to continue, or q <return> to quit---
type=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#37 _PyObject_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#38 0x00005618d2f175c6 in call_function (kwnames=0x7f3239f77410,
oparg=<optimized out>, pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#39 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3139
#40 0x00005618d2e5abda in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#41 0x00005618d2e5ba35 in _PyFunction_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:376
#42 0x00005618d2e5bdd6 in _PyObject_FastCallDict ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
#43 0x00005618d2e79e03 in _PyObject_Call_Prepend ()
---Type <return> to continue, or q <return> to quit---
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
#44 0x00005618d2eb05fa in slot_tp_init ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#45 0x00005618d2eb1208 in type_call (kwds=0x0, args=0x7f31f805c450,
type=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#46 _PyObject_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#47 0x00005618d2f16a16 in call_function (kwnames=0x0, oparg=<optimized out>,
pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#48 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#49 0x00005618d2ea975b in function_code_fastcall (globals=<optimized out>,
nargs=3, args=<optimized out>, co=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:283
---Type <return> to continue, or q <return> to quit---
#50 _PyFunction_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#51 0x00005618d2f124a0 in call_function (kwnames=0x0, oparg=<optimized out>,
pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#52 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3110
#53 0x00005618d2ea975b in function_code_fastcall (globals=<optimized out>,
nargs=0, args=<optimized out>, co=<optimized out>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:283
#54 _PyFunction_FastCallKeywords ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#55 0x00005618d2f12216 in call_function (kwnames=0x0, oparg=<optimized out>,
pp_stack=<synthetic pointer>)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#56 _PyEval_EvalFrameDefault ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/ceval.c:3124
#57 0x00005618d2e5a8f9 in _PyEval_EvalCodeWithName ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#58 0x00005618d2e5b824 in PyEval_EvalCodeEx ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3959
#59 0x00005618d2e5b84c in PyEval_EvalCode (co=co@entry=0x7f3241dee8a0,
globals=globals@entry=0x7f3241e241e0, locals=locals@entry=0x7f3241e241e0)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:524
#60 0x00005618d2f71684 in run_mod ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:1035
#61 0x00005618d2f7bab1 in PyRun_FileExFlags ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:988
#62 0x00005618d2f7bca3 in PyRun_SimpleFileExFlags ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:429
#63 0x00005618d2f7bd6d in PyRun_AnyFileExFlags ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:84
---Type <return> to continue, or q <return> to quit---
#64 0x00005618d2f7cd95 in pymain_run_file (p_cf=0x7ffc94cc9950,
filename=0x5618d4c77980 L"/reg/neh/home/cpo/git/lcls2/install/bin/hsdpva",
fp=0x5618d4cd38c0)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:428
#65 pymain_run_filename (cf=0x7ffc94cc9950, pymain=0x7ffc94cc9a60)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:1607
#66 pymain_run_python (pymain=0x7ffc94cc9a60)
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:2868
#67 pymain_main ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3029
#68 0x00005618d2f7cebc in _Py_UnixMain ()
at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3064
#69 0x00007f32416e8c05 in __libc_start_main () from /lib64/libc.so.6
#70 0x00005618d2f21bd0 in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb)

  • No labels