Discussion on Jan. 18, 2022

with claus, weaver, cpo

Matt's proposal:

• timing on LC, but still handled by BOS
• networking might be on LC as well (unless 40Gb or 100Gb, which is natively MPO). NOT handled by BOS (also XTCAV point-to-point)
• pgp is on MPO and handled by the BOS
• detectors are all LC going to the BOS
• BOS-to-208-patch-panel is MPO (via breakout cable) and some LC as described above
• don't need breakout cables in SRCF according to Matt's proposal (wasteful of fibers between 208 and SRCF?)
• try to use the SFP-to-QSFP-adapters (LC to MPO) as much as possible in SRCF

this is an argument for "specialized" cabling on the SRCF side:

• if we had enough fibers for all SRCF nodes we would hook up MPO cables for all of them
• but we only have 300 fibers pairs to SRCF where ideally we would like 56nodes*8=448 fiber pairs (also need networking fibers)
• also, the bos can only handle 160 fiber pairs going to SRCF
• also, some nodes only need 1 fiber (timing, hsd) so hooking up all fibers is a waste
• it's hard to change what fibers a node needs because we have to change:
  • firmware
  • transceiver (LR4 (hsd), PLR4 (wave8, epixHR), SFP to QSFP converters (timing))order:

• bos has 320 pairs, but only half of these are used for SRCF, the other half goes to detectors (LC).  Implies 40 breakout cables with 4 fiber-pairs per MPO for room 208?
• 56 cmp, 11 eb, 19 mon nodes. currently only cmp have kcu's. maximum of 112 transceivers, but some should be sfp-to-qsfp. hsd's use lr4 (currently 12 channels in TMO and 4 channels in RIX). in feb. 2023 tmo needs 14 for hexanodes plus extra channels (spare and miscellaneous signals).  so guess 25 LR4. expect 36 in TMO eventually.  all others use plr4.  112-25 ~= 90 PLR4 max. Matt guess half will be sfp-to-qsfp.  So 45 PLR4, and 45 sfp-to-qsfp
• 56 cmp nodes corresponds to 56*8=448 fiber pairs (more than the half of the bos devoted to srcf (other half is for detectors))
• order our own patch fibers? (1 foot through 3m or 5m?)
• in short-term need to cable test detectors incrementally
• in long-term consider ripping out and having professionals do it (requires planning).  we do have a short window (2 or 3 weeks?) setup SRCF in advance using estimates of detector numbers. ideally so we can maintain it.
• need to figure out fiber swaps (tx/rx) and how we do it in a neat way. more complex with MPO.  Maybe BOS could do this?  We think not: monitoring might not work.  proposal: best place is from BOS→SRCFpatchpanel. indicate the swap by removing the plastic piece that holds tx/rx in place.

3 sets of limited fiber resources:

• bos: 320 pairs, 160 for detectors, 160 for srcf
• nodes: 56*8=448 pairs
• fibers between 208 and srcf: 300 fiber pairs (48 MPO, 48 LC?).  Total of 240?

SRCF Fiber Order

55
FS QSFP-PLR4-40G Compatible 40GBASE-PLR4 QSFP+ 1310nm 10km DOM MTP/MPO-12 SMF Optical Transceiver Module

 (singlemode plr4 qsfp)
25 
Cisco QSFP-40GE-LR4 Compatible 40GBASE-LR4 QSFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (single mode lr4)
45 
Cisco CVR-QSFP-SFP10G Compatible 40G QSFP+ to 10G SFP+ Adapter Converter Module (sfp to qsfp adapter) CVR-QSFP-SFP10G
60 
Cisco SFP-10G-LR Compatible 10GBASE-LR SFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (singlemode sfp transceiver (for sfp-to-qsfp adapters))
25 
3m (10ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (3m singlemode breakout)
20 
1m (3ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (1m single mode breakout)
45 
Customized 8-144 Fibers Senko MPO-12 OS2 Single Mode Elite Trunk Cable, Yellow (3m mpo12 trunk cable)
20 
0.5m (1.6ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (0.5m patch fiber)
20 
1m (3ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1m patch fiber)
20 
1.5m (5ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1.5m patch fiber)
20 
2m (7ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (2m patch fiber)
20 
3m (10ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (3m patch fiber)
10 
4m (13ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (4m patch fiber)

Sept. 2022 Cabling Proposal

Executive summary: I think at 100kHz we can keep roughly the same number of nodes that we have in the FEE alcove.

Issues:

• which fibers should be MPO vs LC?
  • only use MPO for fim's.
• should we fanout timing at SRCF or put it through the bos?
  • use the fanouts at SRCF for simplicity
  • should we fanout in 208?
  • maybe put camlink data fibers through the bos?
• routing mono-encoder udp network over to srcf
• sharing xtcav between hutches with bos-switchable timing

100kHz data volumes:

• wave8: 8waveforms*256samples/sample*2bytes/sample*0.1MHz = 400MB/s (need to add FEX data too)
• piranha: 2048px*2bytes/px*0.1MHz=410MB/s
• hsd full wf: 60000samples*2bytes/sample*0.1MHz = 12GB/s
  • max waveform length appears to be ~60000 samples looking at xtc2 file?
  • FEX needs to reduce it to 4GB/s (3x)
• bld: 336bytes/event*0.1MHz = 33MB/s (from xtcreader -f /cds/data/psdm/prj/public01/xtc/tmoc00118-r0222-s008-c000.xtc2)
• timing: 309bytes/event*0.1MHz = 31MB/s

summary:

• hook it up similarly to the way it is done in the FEE alcove, but more neatly
• fanout the timing in 208 and put everything through the BOS

recabling proposal:

• tmo timing (including pvadet, bld) (LC, SFP to QSFP)
• rix timing (LC, SFP to QSFP)
• (4) tmo camlink nodes (LC, needs timing)
• (2) rix camlink nodes (LC, needs timing)
• tmo/rix fim node (5 fims) (MPO, PLR4)
• (2) rix hsd nodes (2 LC per node, LR4)
• (7) tmo hsd nodes (2 LC per node, LR4)
• peppex hsd node (LC LR4)
• (2) tmo/rix ami
• (2) tmo/rix teb
• (2) tmo/rix mebuser
• xtcav (LC, needs timing)
• mono-encoder (LC, needs timing)
• in SRCF:
  • let's try MPO for everything (since we only have 48 LC)
  • tdet (12, timing): SFP fiber (from an MPO breakout cable) SFP to QSFP converter
    • 3 MPO breakout, 12 SFP to QSFP converters, 12 SFP transceivers
  • ilv (10, hsd): SFP fiber with LR4 transceiver (no MPO)
    • 5 MPO breakout, 20 LR4
  • wave8 (1, XilinxKcu1500Pgp4_6Gbps_ddr): MPO fiber with PLR4 transceiver
    • 1 MPO, 1 PLR4
  • camlink (4, ClinkKcu1500Pgp2b): data (transceiver 0, up to 4 lanes/cameras) MPO fiber with PLR4, timing (transceiver 1), try MPO fiber with PLR4 (neater, but wastes fibers)
    • 8 MPO, 8 PLR4

current inventory (sept. 13, 2022):

• 27 nodes needed
• shutdown 30
• 3 broken (cmp006, 13, 15)

SRCF cabling

Nodes

Cabling

Top Patch Panel:

NOTE: the rightmost MPO connections are MPO12 (2 rows, 4 columns).  The left ones are MPO8.  We should use MPO8 for KCU.  The MPO12 are reserved for "future expansion" (can be converted to MPO8, I think).

Bottom Patch Panel (MPO):

Bottom Patch Panel (LC):

LC

Fibers 15-16: XTCAV (SXR, cpo believes) from Room 208 Rack 2 PH2 fibers 61-62 → cmp001.  61/62 means sixth cassette from the left, top fiber.

Fibers 21-22: MonoEncoder from ATCA Switch in FEE → cmp002

TMO Hutch Fiber Connections

RIX Hutch Fiber Connections

MFX Hutch Fiber Connections

XTCAV Fiber Route

SXR

From Matt: I tried to look up the exact path of the xtcav fiber, but couldn't trace it all the way through captar.  I know it comes from B921 Rack 73 to B921 BDF1;  Chris and I traced this fiber.  From there it goes to B005 Rack 611 and then to B950-R42 (now renumbered as R02) at elevation 36, I believe.  There are lots of fibers at that location, so we may need two people to locate it - hold a strobe to one end and look for it on the other end.

And rougher notes that cpo doesn't currently understand:

rack B921-BDF1
SM slot 3 to B005 FODU 3 RK 611
fibers 3 4 in fodu

In building 921 rack 73 the DMPH (HXR) server is at height 24 and DMPS (SXR) server is at height 22.

HXR

TBD, but presumably similar to SXR.

Pictures of B921 Rack 73

Front and back

Room 208 Fiber Connections

See FEE Alcove for description of which patch panel in 208 goes to which detector.  Note that the fiber numbering on that page is different from here.  On the FEE alcove page we number fibers left-to-right, then top-to-bottom.  Here we use a better scheme: using the actual fiber numbers marked on the cassettes.  We should convert the FEE-alcove page to use the better fiber-numbering scheme.

XPM Topology

lcls2Timing -> xpm0 -> 2 (tmo) -> 4 (tmo)
                               -> 6 (208) -> BOS
                    -> 3 (rix) -> 5 (208) -> BOS
                    -> 10(fee) -> 11(fee)
lcls1Timing -> xpm7

Problematic connections SRCF ↔ 208

• DRP-SRCF-CMP018-QSFP1 → 5.3.6 (Shows -7.27, is an HSD)
• DRP-SRCF-CMP008 (reported by ric):  Connecting DRP-SRCF-CMP008.0_TIMING (5.5.1) to XPM-6_AMC-0_SFP-1 (5.8.4) doesn’t bring up ‘TDetSim/cmp008’ in xpmpva and shows LinkRxErrs
• move base processes to mon001

Cleanup:

• old neh cmp013 going to xpm3 amc0 port 5
• old neh cmp015 going to xpm3 amc1 port 0
• LC patch panel connections to SRCF
• leftover long fibers in 208