Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • 27 nodes needed
  • shutdown 30
  • 3 broken (cmp006, 13, 15)

SRCF cabling

Nodes

QSFP 1 cage is not workingNo KCU
drp-srcf-cmpNNNKCU firmwareCommentApplication (Oct 17, 2022)
001DrpTDetTMO low rate, XTCAV nic (SXR, cpo believes)slow tmo tdet (+xtcav (SXR, cpo believes))
002DrpTDetRIX low rate, encoder nicslow rix tdet (+mono)
003DrpTDetTMO high ratetmo high rate tdet
004XilinxKcu1500Pgp4_6Gbps_ddr
rix/tmo wave8
Collection host
005DrpPgpIlv
rix hsd's
006ClinkKcu1500Pgp2b LCLSECSD-1844.  Possibly repaired spare camlink?spare camlink node?
007ClinkKcu1500Pgp2b
tmo fzp piranha
008DrpTDetdidn't show up on xpmrix user meb
009DrpPgpIlv
rix hsd's
010DrpTDetRIX high raterix fast tdet
011ClinkKcu1500Pgp2b
tmo fzp opal
012ClinkKcu1500Pgp2b
tmo opal1 rix piranha
013ClinkKcu1500Pgp2b
tmo atm piranha
014Lcls2EpixHrXilinxKcu1500Pgp4_6Gbps
epix hr
015
No mlx5; Bad Ethernet
LCLSECSD-1305
Unreliable machine: bad for TEB / MEB
nothing
016DrpTDet017DrpPgpIlv018DrpPgpIlv019DrpPgpIlv020DrpPgpIlv021DrpPgpIlv022DrpPgpIlv023DrpPgpIlv024DrpPgpIlv025DrpTDet026DrpTDet027DrpTDet028DrpTDet029DrpTDet030DrpTDet

Cabling

Top Patch Panel:

...

cmp006 (timing) (close to the USB port)

...

1: cmp022(away from USB port)

2: cmp022(close to USB port)

3: cmp023 (away from USB port)

4: cmp023 (close to USB port)

...

1: cmp018(away from USB port)

2: cmp018(close to USB port)

3: cmp019 (away from USB port)

4: cmp019 (close to USB port)

...

1: cmp009(away from USB port)

2: cmp009(close to USB port)

3: cmp017 (away from USB port)

4: cmp017 (close to USB port)

...

1: cmp008

2: cmp010

3: cmp016

4: cmp025

...

1: cmp024(away from USB port)

2: cmp024(close to USB port)

3: -

4: -

...

1: cmp020(away from USB port)

2: cmp020(close to USB port)

3: cmp021 (away from USB port)

4: cmp021 (close to USB port)

...

1: cmp029

2: cmp030

3: -

4: -

...

1: cmp026

2: cmp027

3: cmp028

4: -

Room 208 Fiber Connections

See FEE Alcove Fiber Cabling for description of which patch panel in 208 goes to which detector.  Note that the fiber numbering on that page is different from here.  On the FEE alcove page we number fibers left-to-right, then top-to-bottom.  Here we use a better scheme: using the actual fiber numbers marked on the cassettes.  We should convert the FEE-alcove page to use the better scheme.

...

XPM Topology

...

No mlx5.  Error messages
LCLSECSD-1305
Unreliable machine: bad for TEB / MEB
nothing
017DrpPgpIlv
tmo hsd_3
018DrpPgpIlv
tmo hsd_5,hsd_7
019DrpPgpIlv
tmo hsd_4,hsd_13
020DrpPgpIlv
tmo hsd_2
021DrpPgpIlv
tmo hsd_6
022DrpPgpIlv
tmo hsd_8,hsd_9
023DrpPgpIlv
peppex hsd's
024DrpPgpIlv
tmo hsd_10,hsd_11
025DrpTDet
rix slow tdet
026ClinkKcu1500Pgp2b
tmo atm opal
027ClinkKcu1500Pgp2b
rix atm opal
028DrpTDetXPM:6tmo ami
029DrpTDetXPM:5rix ami
030DrpTDetNCtmo user meb
031XilinxKcu1500Pgp4_6Gbps_ddrLCLSECSD-2030: BIOS not upgradabletmo user meb
032DrpTDetXPM:7 (hacked with fiber from xpm0 that was in cmp037).  Web IPMI interface broken.  Update bios settings: only one pcie bus.LCLS1 HXR hutches
033DrpTDetXPM:0.  update bios settings: only one pcie bus.
034DrpTDetXPM:0mona slurm tests
035DrpTDetXPM:0; LCLSECSD-2032: BIOS  not upgradable from v2.0 but is useable as-ismona slurm tests
036DrpTDetepixM dataepixM
037DrpTDetXPM:0 (hacked with fiber from xpm7)
038DrpTDetXPM:7
039DrpTDet LCLSECSD-2028: Broken IPMI
040DrpTDetXPM:0
041DrpTDetXPM:0
042DrpTDetXPM:0
043DrpTDetXPM:0EpixHrEmu
044DrpTDetXPM:0Laser hall
045Todo: XilinxKcu1500Pgp4_6GbpsXPM:0TXI Wave8
mon001

tcmp03mo meb and ami_manager

neh-base.cnf, hsd.cnf, rix-hsd.cnf

046DrpPgpIlv

TMO HSD_15 and TMO HSD_16

048DrpPgpIlv

TMO HSD_17 and TMO HSD_18

049DrpPgpIlvnot working, possibly defective KCU


050DrpPgpIlv

TMO HSD_01 and TMO HSD_19

drp-srcf-gpuNNNKCU firmwareCommentApplication
001in progressAdding kcu1500
002n/a

003n/a

004n/a

Cabling

Top Patch Panel:

NOTE: the rightmost MPO connections are MPO12 (2 rows, 4 columns).  The left ones are MPO8.  We should use MPO8 for KCU.  The MPO12 are reserved for "future expansion" (can be converted to MPO8, I think).

(6-strand breakout cable)
1: gpu001 (away from usb port)
2: cmp040 (away from usb port)
3: cmp041 (away from usb port)
4: cmp042 (away from usb port)
5: cmp043 (away from usb port)
6: cmp044 (away from usb port)




DON'USE FOR NOWDON'T USE FOR NOW



(breakout cable)

1: cmp022(away from USB port)

2: cmp022(close to USB port)

3: cmp023 (away from USB port)

4: cmp023 (close to USB port)

(breakout cable)

1: cmp018(away from USB port)

2: cmp018(close to USB port)

3: cmp019 (away from USB port)

4: cmp019 (close to USB port)

(breakout cable)

1: cmp009(away from USB port)

2: cmp009(close to USB port)

3: cmp017 (away from USB port)

4: cmp017 (close to USB port)

(breakout cable)

1: cmp008

2: cmp010

3: cmp016

4: cmp025

(break cable)
1: cmp033 (away from usb port)
2: cmp035 (away from usb port)
3: cmp036 (away from usb port)
4: cmp032 (away from usb port)






(breakout cable)

2: cmp046 (away from usb)
3: cmp048 (close to usb)
4: cmp048 (away from usb)

(breakout cable)

1: cmp024(away from USB port)

2: cmp024(close to USB port)

3: -

4: -

(breakout cable)

1: cmp020(away from USB port)

2: cmp020(close to USB port)

3: cmp021 (away from USB port)

4: cmp021 (close to USB port)

(breakout cable)

1: cmp029

2: cmp030

3: cmp037

4: cmp038

(breakout cable)

1: cmp028

2: -

3: -

4: -








(breakout cable)
1: cmp050(close to usb)
2: cmp050(away from the usb)
3: cmp046 (close to usb)





Bottom Patch Panel (MPO):

cmp006 (timing) (close to the USB port)cmp011 (timing) (close to the USB port)cmp012 (timing) (close to the USB port)cmp004(close to usb port)cmp013(timing) (close to the USB port)cmp026(timing) (close to the USB port)cmp027(timing) (close to the USB port)cmp004(away from usb port)
cmp006 (data) (away from USB port)cmp011 (data) (away from USB port)cmp012 (data) (away from USB port)cmp013 (data) (away from USB port)cmp045 (away from USB port)cmp026 (data) (away from USB port)cmp027 (data) (away from USB port)cmp014(away from usb port)

Bottom Patch Panel (LC):

IT
ITcmp007 (close to the USB port)

cmp001 (away from the USB port)


ITcmp007 (away from the USB port)

cmp002 (away from the USB port)


IT


cmp003 (away from the USB port)


IT


cmp005 (close to the USB port)



IT


cmp005 (awy from the USB port)


IT



LC

Fibers 15-16: XTCAV (SXR, cpo believes) from Room 208 Rack 2 PH2 fibers 61-62 → cmp001.  61/62 means sixth cassette from the left, top fiber.

Fibers 21-22: MonoEncoder from ATCA Switch in FEE → cmp002

TMO Hutch Fiber Connections

SourceDestinationNotes
XPM 2 input on RTMRack 4 in TMO hutch, FODU2, Cassette 4, fibers 3-4 (a.k.a TMO2.4.3-4)Note that cassettes 3 and 4 are swapped between rack 3/4
XPM 2 network connectionRack 3, Cassette 3, fibers 23-24, goes to Rack 4 FODU 5, Cassette 4, fibers 23-24, goes to Rack 4 FODU 2 cassette 4 fibers 5-6Note that cassettes 3 and 4 are swapped between rack 3/4

RIX Hutch Fiber Connections

SourceDestinationNotes
XPM 3 network connectionRack 1 FODU Cassette 1 fibers 15-16Don't immediately know what the official fodu number is
XPM 3 AMC1 port 5 counting from 0Room 208 RIX1 fodu, cassette 1 fibers 11-12Wasn't connected on the room 208 side on Oct. 26, 2022.  Maybe not necessary any more?

MFX Hutch Fiber Connections

SourceDestinationNotes
BOS 1.8.1, 1.8.2 and XPM7 outputBuilding B950 RB02 in room 208: SM S04 To B999 MEZZ FODU 43 R11, Ports 1,2,43 fiber pairs for epixHR data (two pairs) and timing (one pair)

XTCAV Fiber Route

SXR

From Matt: I tried to look up the exact path of the xtcav fiber, but couldn't trace it all the way through captar.  I know it comes from B921 Rack 73 to B921 BDF1;  Chris and I traced this fiber.  From there it goes to B005 Rack 611 and then to B950-R42 (now renumbered as R02) at elevation 36, I believe.  There are lots of fibers at that location, so we may need two people to locate it - hold a strobe to one end and look for it on the other end.

And rougher notes that cpo doesn't currently understand:

rack B921-BDF1
SM slot 3 to B005 FODU 3 RK 611
fibers 3 4 in fodu

In building 921 rack 73 the DMPH (HXR) server is at height 24 and DMPS (SXR) server is at height 22.

HXR

TBD, but presumably similar to SXR.

Pictures of B921 Rack 73

Front and back

Image AddedImage Added

Room 208 Fiber Connections

See FEE Alcove for description of which patch panel in 208 goes to which detector.  Note that the fiber numbering on that page is different from here.  On the FEE alcove page we number fibers left-to-right, then top-to-bottom.  Here we use a better scheme: using the actual fiber numbers marked on the cassettes.  We should convert the FEE-alcove page to use the better fiber-numbering scheme.

SourceDestinationNotes
ATCA FEE router port 0ATCA room 208 router port 1 10Gb network to SRCF node, the connection is done via the octopus fiber #4
room 208 ATCA crate switch optical fiber port 0SRCF LC cassette 4 (counting from 1, left to right) fibers 11/1210Gb network to SRCF node (drp-srcf-cmp004?)
TMO XPM 2 network connection (TMO FODU 1, Cassette 4 fibers 5-6)ATCA switch optical port 7 counting from 0needed to swap tx/rx fibers to get link-lock
RIX XPM 3 network connection (RIX FODU, cassette 1, fibers 15-16)ATCA switch optical port 6 counting from 0
Mono Encoder NetworkFrom FEE Fodu 1 cassette 3 fibers 1,2 to SRCF LC fodu fibers 21,22
Timing output of XPM2 in 208TMO FODU 1, Cassette 3, Fibers 17-18: 1.3.17-18Cassette 1 is missing
Timing output of XPM3 in 208RIX FODU 1, Cassette 1, Fibers 7-8: 1.1.7-8
XPM0-AMC0-PORT0RIX 1.1.3-4This is XPM0 → XPM3 in RIX, FODU 1 for RIX is FODU 5 in 208
XPM0-AMC0-PORT1TMO 2.4.3-4This is XPM0 → XPM2 in TMO, FODU 2 for TMO is FODU 7 in 208
XPM0-AMC1-PORT2TMO 2.3.33-34 (tmo cassette 3 fiber 33-34)Backup connection for XPM0->XPM2
XPM5-AMC0-PORT0RIX 1.1.3-4This is XPM3→ XPM5
XPM6-AMC0-PORT0TMO 1.3.17-18This is XPM2→ XPM6
XPM0-AMC1-PORT0FEE FODU 1, Cassette 4 fiber 11-12 (top right)LCLS2 timing for FEE Teststand
TMO ATM CAMLINKTMO FODU 1 cassette 3 fiber 1-2 (a.k.a. TMO1.3.1-2)cassette 1 is missing, swapped tx/rx at bos
RIX FIM MR4K2RIX FODU 1 cassette 1 fibers 25-26 (a.k.a RIX1.1.25-26)
RIX FIM MR3K2RIX1.1.27-28
RIX FIM CHEMRIXRIX1.1.29-30
TMO FIM1TMO1.3.5-6Not sure which FIM this really corresponds to in TMO
TMO FIM2TMO1.3.7-8Not sure which FIM this really corresponds to in TMO
RIX HSD2RIX1.1.19-20I may be swapping hsd2/3 (this was FEE fodu 1.3.18, old numbering scheme)
RIX HSD3RIX1.1.23-24I may be swapping hsd2/3 (this was FEE fodu 1.3.15, old numbering scheme)
TMO CAMLINK OPAL1TMO1.4.1-2swapped tx/rx at bos
TMO CAMLINK OPAL2TMO1.3.3-4swapped tx/rx at bos
TMO CAMLINK FZPTMO1.2.11-12swapped tx/rx at bos
RIX CAMLINK ATMRIX1.3.13-14
TMO HSD3 or companionTMO1.4.13-14I may be swapping hsd3 with its companion (don't know the name: not in tmo.cnf?).  This one was in fee fodu1.1.11 (old numbering scheme).  Needed tx/rx swap at bos.
TMO HSD3 or companionTMO1.4.15-16I may be swapping hsd3 with its companion (don't know the name: not in tmo.cnf?).  This one was in fee fodu1.1.9 (old numbering scheme).  Needed tx/rx swap at bos.
TMO HSD5 or HSD7TMO1.4.31-32I may be swapping hsd5/hsd7.  This one was in fee fodu 1.1.1 (old numbering scheme).  Needed tx/rx swap at bos.
TMO HSD5 or HSD7TMO1.4.29-30I may be swapping hsd5/hsd7.  This one was in fee fodu 1.1.5 (old numbering scheme).  Needed tx/rx swap at bos.
TMO HSD4TMO1.4.27-28Was in fee fodu 1.1.4 (old numbering scheme). tx/rx swap at bos.
TMO HSD4 companionTMO1.4.33-34Was in fee fodu 1.1.6 (old numbering scheme).  tx/rx swap at bos.
TMO HSD1TMO1.4.17-18Was in fee fodu 1.1.12 (old numbering scheme).  tx/rx swap at bos.
TMO HSD2TMO1.4.19-20Was in fee fodu 1.1.8 (old numbering scheme).  NO tx/rx swap at bos!
TMO HSD6TMO1.4.35-36Was in fee fodu 1.1.3 (old numbering scheme).  tx/rx swap at bos.
TMO HSD6 companionTMO1.4.25-26Was in fee fodu 1.1.2 (old numbering scheme).  tx/rx swap at bos.
TMO HSD9TMO1.4.21-22Was in fee fodu 1.1.7 (old numbering scheme).  tx/rx swap at bos.
TMO HSD8TMO1.4.23-24Was in fee fodu 1.1.10 (old numbering scheme).  tx/rx swap at bos.
RIX HSD1RIX1.1.21-22Was in fee fodu 1.3.8 (old numbering scheme)
RIX HSD0RIX1.1.17-18Was in fee fodu 1.3.9 (old numbering scheme)
PEPPEX HSD1Room 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 21-22 (second from bottom).  Also labelled "slot 4" on the slot itself.Was in fee fodu 2.4.3 (old numbering scheme)
PEPPEX HSD0Room 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 19-20 (third from bottom). Also labelled "slot 4" on the slot itself.Was in fee fodu 2.4.4 (old numbering scheme)
PEPPEX OPALRoom 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 23-24 (bottom pair). Also labelled "slot 4" on the slot itself.Was in fee fodu 2.4.5 (old numbering scheme)
XTCAV (SXR, cpo believes)Room 208 Rack 2 PH2 fibers 61-62 from B005 (each cassette has 12 fibers, so top of sixth cassette from left)Was in fee fodu 1.2.2 (old numbering scheme)
TXI FIM

Room 208

XPM 5 AMC 1 SFP 2 (20240112) XPM 6 AMC 1 SFP 6 To FODU 8 Cassete 2 Port 9-10

Data To FODU 8 Cassete 2 Port 11-12 


TMO LCLS1 timing

XPM7 AMC1 port1 to TMO FODU 7 Cassette 3 (counting from 1 with 4 cassettes total, one currently unoccupied) fibers 11-12


XPM0-AMC1-PORT4FEE FODU 6, Cassette 3 fiber 23-24 (lower right)LCLS2 timing for FEE Teststand XPM10 RTM
TMO HSD 88 or 89BOS (2.1)1-2 or 5-6In TMO Hutch connected to FODU1 Cassette 1 position  8 (most left in the middle row might be broken) and  15  (lowest row int he middle) RACK 4
in 208 FODU 7 RACK 4
TMO HSD 89 or 88BOS(2.1) 5-6 or 1-2in TMO Hutch connected to FODU1 Cassette 1 position 9 and 10 RACK 4
in 208 FODU 7 RACK 4
N.B. position 9 (in the middle row third from the left) was already taken in 208 but not in TMO. Connected to  FODU4 (208) cassette 2 (second from the left) position 7 (first from the left in the middle), but removed because no fiber in TMO.
TMO HSD 1 (1B:B)BOS 1.8.3In TMO hutch connected to FODU1 Cassette 1 position 3 (counting from top left from 1) RACK4
TMO HSD 18 (B1:B)BOS 1.8.4In TMO hutch connected to FODU1 Cassette 1 position 4 (counting from top left from 1) RACK4
RIX High-rate encoder in FEE (data and timing, in that order)

From FEE: B940-008-R03-FODU-U1 PAIRS 2 and 3 COUNTING FROM 1

Through FEE alcove: B940-009-R06-FOD3-U1 pairs 2 and 5 counting from 1

To B950-208-r42-FOD3-U2 PAIRS 2 and 5 COUNTING FROM 1

Swapped tx/rx at xpm5 amc1 port 2 (counting from 0).

Data also needed to be swapped going into BOS (1.7.8).


XPM Topology

Code Block
lcls2Timing -> xpm0 -> 2 (tmo) -> 4 (tmo)
                               -> 6 (208) -> BOS
                    -> 3 (rix) -> 5 (208) -> BOS
                    -> 10(fee) -> 11(fee)
lcls1Timing -> xpm7

Problematic connections SRCF ↔ 208

  • DRP-SRCF-CMP018-QSFP1 → 5.3.6 (Shows -7.27, is an HSD)
  • DRP-SRCF-CMP008 (reported by ric):  Connecting DRP-SRCF-CMP008.0_TIMING (5.5.1) to XPM-6_AMC-0_SFP-1 (5.8.4) doesn’t bring up ‘TDetSim/cmp008’ in xpmpva and shows LinkRxErrs
  • move base processes to mon001

Cleanup:

  • old neh cmp013 going to xpm3 amc0 port 5
  • old neh cmp015 going to xpm3 amc1 port 0
  • LC patch panel connections to SRCF
  • leftover long fibers in 208

...