Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

UNSOLVED: got a segv in this log file: /reg/neh/home/cpo/2020/06/15_18:39:03_daq-tst-dev03.pcdsn:hsdpva.log.  restarting the process fixed it.  also here:  Later, generated a core corresponding to this log /reg/neh/home/cpo/2020/06/16_15_18:5449:0013_daq-tst-dev03.pcdsn:xpmpvahsdpva.log.  Later, generated a core:

Code Block
(ps-3.1.11) daq-tst-dev03:lcls2$ gdb `which python` ./psdaq/psdaq/cnf/core.hsdpva.23738
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/bin/python3.7...done.
warning: core file may not match specified executable file.
[New LWP 23738]
[New LWP 24583]
[New LWP 24585]
[New LWP 24580]
[New LWP 24589]
[New LWP 24579]
[New LWP 24630]
[New LWP 24582]
[New LWP 24581]
[New LWP 24584]
[New LWP 24615]
[New LWP 25101]
[New LWP 24619]
[New LWP 25163]
[New LWP 25156]
[New LWP 25099]
[New LWP 24617]
[New LWP 25716]
[New LWP 25441]
[New LWP 25758]
[New LWP 25160]
[New LWP 25783]
[New LWP 25771]
[New LWP 25784]
[New LWP 25803]
[New LWP 25798]
[New LWP 25757]
[New LWP 25772]
[New LWP 25155]
[New LWP 25752]
[New LWP 25715]
[New LWP 25801]
[New LWP 25799]
[New LWP 25440]
[New LWP 25751]
[New LWP 25787]
[New LWP 25782]
[New LWP 24629]
[New LWP 24634]
Missing separate debuginfo for /lib64/libpthread.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/60/91139b61d901aa8425dbc16b7f51d455a416b4.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /lib64/libc.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ff/1d2de4cf353118960c5ea195ceb13699929c14.debug
Missing separate debuginfo for /lib64/libdl.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/ef/023e0c16991e8957337c90b0e51025fe27c897.debug
Missing separate debuginfo for /lib64/libutil.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b8/082f4480af5af680a0ad0ab4818cae1c077a0b.debug
Missing separate debuginfo for /lib64/librt.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/76/c4765826961c60be913f5549ebbc7bbe506899.debug
Missing separate debuginfo for /lib64/libm.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/6a/344a5ce77a247399aa2f22f92c698f574a7134.debug
Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a9/980cf253c79740e69f70dcb8fea7b8c2f641b5.debug
Missing separate debuginfo for /lib64/libGL.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a2/f537e16380004afe5920223ba4837a7cf9193f.debug
Missing separate debuginfo for /lib64/libxshmfence.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/09/4eafa9d4305b9d112a97d33a68361fe72073b7.debug
Missing separate debuginfo for /lib64/libglapi.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/74/370865c18663a14d50292b39ba4589a049e4e7.debug
Missing separate debuginfo for /lib64/libselinux.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/6a/5ef7a05f7e488fcd280bfadd96083bec9fd416.debug
Missing separate debuginfo for /lib64/libXdamage.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f2/709bb30b8496f436775e13d2250d5565fa8b64.debug
Missing separate debuginfo for /lib64/libXfixes.so.3
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/062d2deb4d533956a51bcf82f487e2bcd14c11.debug
Missing separate debuginfo for /lib64/libXxf86vm.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/56/9cf7657010c08e28262476726d80eb9bb8ea02.debug
Missing separate debuginfo for /lib64/libdrm.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/37/7b9e3631cf45fa052067dce55171b3c0340bfa.debug
Missing separate debuginfo for /lib64/libXcursor.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/27/ce4043698294568faa1373d44a82f74e49f08b.debug
Core was generated by `/reg/g/psdm/sw/conda2/inst/envs/ps-3.1.11/bin/python /reg/neh/home/cpo/git/lcls'.
Program terminated with signal 11, Segmentation fault.
#0  0x00005618d53cb360 in ?? ()
(gdb) where
#0  0x00005618d53cb360 in ?? ()
#1  0x00005618d2da2c5f in visit_decref (op=<optimized out>, 
    data=<optimized out>, op=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:306
#2  0x00005618d2e3bde7 in list_traverse (o=0x7f31f805ad70, 
    visit=0x5618d2e3fc40 <visit_decref>, arg=0x0)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/listobject.c:2583
#3  0x00005618d2e68f74 in subtract_refs ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:296
#4  collect ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:853
#5  0x00005618d2e69a24 in collect_with_callback ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1025
#6  0x00005618d2f2f939 in collect_generations ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1048
#7  0x00005618d2e69b84 in _PyObject_GC_Alloc (basicsize=<optimized out>, 
    use_calloc=0)
---Type <return> to continue, or q <return> to quit---
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1705
#8  _PyObject_GC_Malloc ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1715
#9  0x00005618d2e69bae in _PyObject_GC_New (
    tp=tp@entry=0x5618d3083ce0 <PyMethod_Type>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/gcmodule.c:1727
#10 0x00005618d2e6dd92 in PyMethod_New ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/classobject.c:62
#11 0x00005618d2ea6cf0 in super_getattro ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:7570
#12 0x00005618d2f08c0c in _PyObject_GetMethod ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/object.c:1137
#13 0x00005618d2f1238e in _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3040
#14 0x00005618d2e5abda in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/ceval.c:3930
#15 0x00005618d2ea99e7 in _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
#16 0x00005618d2f124a0 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>13782
 
(gdb) where
#0  0x00007ff0942ed1f7 in raise () from /lib64/libc.so.6
#1  0x00007ff0942ee8e8 in abort () from /lib64/libc.so.6
#2  0x00005631b9a7ced3 in fatal_error (prefix=prefix@entry=0x0, 
    msg=msg@entry=0x5631b9c4f5b9 "GC object already tracked", 
    status=status@entry=-1)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/cevalpylifecycle.c:4616
#17 _PyEval_EvalFrameDefault (2187
#3  0x00005631b9a7cf34 in Py_FatalError (
    msg=msg@entry=0x5631b9c4f5b9 "GC object already tracked")
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/cevalpylifecycle.c:31102197
#18#4  0x00005618d2e5b0b80x00005631b9a84b0b in _PyEvalPyTuple_EvalCodeWithNameNew (size=size@entry=2)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/PythonObjects/cevaltupleobject.c:3930133
#19#5  0x00005618d2e5bc600x00005631b9b3bb85 in _PyFunctionPyStack_FastCallDictAsTuple (stack=stack@entry=0x5631bbaecb00, 
    nargs=nargs@entry=2)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:3761284
#20#6  0x00005618d2e5bdd60x00005631b9b82149 in _PyObject_FastCallDictFastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98177
#21#7  0x00005618d2e79e030x00005631b9be7a16 in _PyObject_Call_Prepend ()call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
---Type <return> to continue, or q <return> to quit---
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/ObjectsPython/callceval.c:908
---Type <return> to continue, or q <return> to quit---
#22 0x00005618d2eb05fa in slot_tp_init4619
#8  _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/ObjectsPython/typeobjectceval.c:66363124
#23#9  0x00005618d2eb12080x00005631b9b2bbda in type_PyEval_callEvalCodeWithName (kwds=0x7f31bc7ba820, args=0x7f32024c1b90, 
    type=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/ObjectsPython/typeobjectceval.c:971
#243930
#10 0x00005631b9b7a9e7 in _PyObjectPyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199433
#25#11 0x00005618d2f175c60x00005631b9be34a0 in call_function (kwnames=0x7f32024fe2500x0, 
oparg=<optimized out>, 
  oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:46194616
#26#12 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:31393110
#27#13 0x00005618d2e5a8f90x00005631b9b2c0b8 in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#28#14 0x00005618d2ea99850x00005631b9b2cc60 in _PyFunction_FastCallKeywordsFastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:433
Obje---Type <return> to continue, or q <return> to quit---
#29 0x00005618d2f12216
cts/call.c:376
#15 0x00005631b9b2cdd6 in call_PyObject_functionFastCallDict (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/PythonObjects/cevalcall.c:4616
#3098
#16 0x00005631b9b4ae03 in _PyEvalPyObject_Call_EvalFrameDefaultPrepend ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/PythonObjects/cevalcall.c:3124908
#31#17 0x00005618d2e5abda0x00005631b9b815fa in slot_PyEvaltp_EvalCodeWithNameinit ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/PythonObjects/cevaltypeobject.c:39306636
#32#18 0x00005618d2e5bc600x00005631b9b82208 in type_PyFunction_FastCallDict (call (kwds=0x7ff04c08cf00, args=0x7ff0550b0c30, 
    type=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/calltypeobject.c:376
#33 0x00005618d2e5bdd6 in971
#19 _PyObject_FastCallDictFastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98199
#34#20 0x00005618d2e79e030x00005631b9be85c6 in _PyObject_Call_Prepend (call_function (kwnames=0x7ff0550ef250, 
    oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/ObjectsPython/callceval.c:9084619
#35#21 0x00005618d2eb05fa in slot_tpPyEval_initEvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#36 0x00005618d2eb1208 in type_call (kwds=0x7f31bc7ad230, args=0x7f320248cc30, 
_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/ceval.c:3139
#22 0x00005631b9b2b8f9 in  type=<optimized out>_PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/ObjectsPython/typeobjectceval.c:971
#373930
#23 0x00005631b9b7a9e7 in _PyObjectPyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199433
#38#24 0x00005618d2f175c60x00005631b9be42e7 in call_function (kwnames=0x7f3239f774100x7ff08cb636d0, 
    oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:46194616
#39#25 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3139
#40#26 0x00005618d2e5abda0x00005631b9b2bbda in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#41#27 0x00005618d2e5ba350x00005631b9b2ca35 in _PyFunction_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:376
#42#28 0x00005618d2e5bdd60x00005631b9b2cdd6 in _PyObject_FastCallDict ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:98
#43 0x00005618d2e79e03 in _PyObject_Call_Prepend ()
---Type <return> to continue, or q <return> to quit----
#29 0x00005631b9b4ae03 in _PyObject_Call_Prepend ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:908
#44#30 0x00005618d2eb05fa0x00005631b9b815fa in slot_tp_init ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:6636
#45#31 0x00005618d2eb12080x00005631b9b82208 in type_call (kwds=0x0, args=0x7f31f805c4500x7ff05507cdc0, 
    type=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/typeobject.c:971
#46#32 _PyObject_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:199
#47#33 0x00005618d2f16a160x00005631b9be7a16 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4619
#48#34 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#49#35 0x00005618d2ea975b0x00005631b9b7a75b in function_code_fastcall (globals=<optimized out>, 
    nargs=3, args=<optimized out>, co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:283
Obje---Type <return> to continue, or q <return> to quit---
#50
cts/call.c:283
#36 _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#51#37 0x00005618d2f124a00x00005631b9be34a0 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#52#38 _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3110
#53#39 0x00005618d2ea975b0x00005631b9b7a75b in function_code_fastcall (globals=<optimized out>, 
    nargs=0, args=<optimized out>, co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:283
#54#40 _PyFunction_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Objects/call.c:408
#55#41 0x00005618d2f122160x00005631b9be3216 in call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:4616
#56#42 _PyEval_EvalFrameDefault ()
---Type <return> to continue, or q <return> to quit---
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Pyth---Type <return> to continue, or q <return> to quit---
on/build_artifacts/python_1585001848288/work/Python/ceval.c:3124
#57#43 0x00005618d2e5a8f90x00005631b9b2b8f9 in _PyEval_EvalCodeWithName ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3930
#58#44 0x00005618d2e5b8240x00005631b9b2c824 in PyEval_EvalCodeEx ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:3959
#59#45 0x00005618d2e5b84c0x00005631b9b2c84c in PyEval_EvalCode (co=co@entry=0x7f3241dee8a00x7ff0949dd8a0, 
    globals=globals@entry=0x7f3241e241e00x7ff094a131e0, locals=locals@entry=0x7f3241e241e00x7ff094a131e0)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/ceval.c:524
#60#46 0x00005618d2f716840x00005631b9c42684 in run_mod ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:1035
#61#47 0x00005618d2f7bab10x00005631b9c4cab1 in PyRun_FileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:988
#62#48 0x00005618d2f7bca30x00005631b9c4cca3 in PyRun_SimpleFileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:429
#63#49 0x00005618d2f7bd6d0x00005631b9c4cd6d in PyRun_AnyFileExFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Python/pythonrun.c:84
Pyth---Type <return> to continue, or q <return> to quit---
#64 0x00005618d2f7cd95
on/pythonrun.c:84
#50 0x00005631b9c4dd95 in pymain_run_file (p_cf=0x7ffc94cc99500x7fff187f5c40, 
    filename=0x5618d4c779800x5631bb10e980 L"/reg/neh/home/cpo/git/lcls2/install/bin/hsdpva", 
    fp=0x5618d4cd38c00x5631bb16a8c0)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:428
#65#51 pymain_run_filename (cf=0x7ffc94cc99500x7fff187f5c40, pymain=0x7ffc94cc9a600x7fff187f5d50)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:1607
#66#52 pymain_run_python (pymain=0x7ffc94cc9a600x7fff187f5d50)
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:2868
#67#53 pymain_main ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3029
#68#54 0x00005618d2f7cebc0x00005631b9c4debc in _Py_UnixMain ()
    at /home/conda/feedstock_root/build_artifacts/python_1585001848288/work/Modules/main.c:3064
#69#55 0x00007f32416e8c050x00007ff0942d9c05 in __libc_start_main () from /lib64/libc.so.6
#70#56 0x00005618d2f21bd00x00005631b9bf2bd0 in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb) 



UNSOLVED: Saw this error in /reg/neh/home/cpo/2020/06/15_14:39:26_daq-tst-dev07:hsdioc_89.log when starting up processes after a power-cycle (for new hsd firmware for the interrupt-holdoff):

...