You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 63 Next »

Discussion on Jan. 18, 2022

with claus, weaver, cpo

Matt's proposal:

  • timing on LC, but still handled by BOS
  • networking might be on LC as well (unless 40Gb or 100Gb, which is natively MPO). NOT handled by BOS (also XTCAV point-to-point)
  • pgp is on MPO and handled by the BOS
  • detectors are all LC going to the BOS
  • BOS-to-208-patch-panel is MPO (via breakout cable) and some LC as described above
  • don't need breakout cables in SRCF according to Matt's proposal (wasteful of fibers between 208 and SRCF?)
  • try to use the SFP-to-QSFP-adapters (LC to MPO) as much as possible in SRCF

this is an argument for "specialized" cabling on the SRCF side:

  • if we had enough fibers for all SRCF nodes we would hook up MPO cables for all of them
  • but we only have 300 fibers pairs to SRCF where ideally we would like 56nodes*8=448 fiber pairs (also need networking fibers)
  • also, the bos can only handle 160 fiber pairs going to SRCF
  • also, some nodes only need 1 fiber (timing, hsd) so hooking up all fibers is a waste
  • it's hard to change what fibers a node needs because we have to change:
    • firmware
    • transceiver (LR4 (hsd), PLR4 (wave8, epixHR), SFP to QSFP converters (timing))order:
  • bos has 320 pairs, but only half of these are used for SRCF, the other half goes to detectors (LC).  Implies 40 breakout cables with 4 fiber-pairs per MPO for room 208?
  • 56 cmp, 11 eb, 19 mon nodes. currently only cmp have kcu's. maximum of 112 transceivers, but some should be sfp-to-qsfp. hsd's use lr4 (currently 12 channels in TMO and 4 channels in RIX). in feb. 2023 tmo needs 14 for hexanodes plus extra channels (spare and miscellaneous signals).  so guess 25 LR4. expect 36 in TMO eventually.  all others use plr4.  112-25 ~= 90 PLR4 max. Matt guess half will be sfp-to-qsfp.  So 45 PLR4, and 45 sfp-to-qsfp
  • 56 cmp nodes corresponds to 56*8=448 fiber pairs (more than the half of the bos devoted to srcf (other half is for detectors))
  • order our own patch fibers? (1 foot through 3m or 5m?)
  • in short-term need to cable test detectors incrementally
  • in long-term consider ripping out and having professionals do it (requires planning).  we do have a short window (2 or 3 weeks?) setup SRCF in advance using estimates of detector numbers. ideally so we can maintain it.
  • need to figure out fiber swaps (tx/rx) and how we do it in a neat way. more complex with MPO.  Maybe BOS could do this?  We think not: monitoring might not work.  proposal: best place is from BOS→SRCFpatchpanel. indicate the swap by removing the plastic piece that holds tx/rx in place.

3 sets of limited fiber resources:

  • bos: 320 pairs, 160 for detectors, 160 for srcf
  • nodes: 56*8=448 pairs
  • fibers between 208 and srcf: 300 fiber pairs (48 MPO, 48 LC?).  Total of 240?

SRCF Fiber Order

55
FS QSFP-PLR4-40G Compatible 40GBASE-PLR4 QSFP+ 1310nm 10km DOM MTP/MPO-12 SMF Optical Transceiver Module

 (singlemode plr4 qsfp)
25 
Cisco QSFP-40GE-LR4 Compatible 40GBASE-LR4 QSFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (single mode lr4)
45 
Cisco CVR-QSFP-SFP10G Compatible 40G QSFP+ to 10G SFP+ Adapter Converter Module (sfp to qsfp adapter) CVR-QSFP-SFP10G
60 
Cisco SFP-10G-LR Compatible 10GBASE-LR SFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (singlemode sfp transceiver (for sfp-to-qsfp adapters))
25 
3m (10ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (3m singlemode breakout)
20 
1m (3ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (1m single mode breakout)
45 
Customized 8-144 Fibers Senko MPO-12 OS2 Single Mode Elite Trunk Cable, Yellow (3m mpo12 trunk cable)
20 
0.5m (1.6ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (0.5m patch fiber)
20 
1m (3ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1m patch fiber)
20 
1.5m (5ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1.5m patch fiber)
20 
2m (7ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (2m patch fiber)
20 
3m (10ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (3m patch fiber)
10 
4m (13ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (4m patch fiber)

Sept. 2022 Cabling Proposal

Executive summary: I think at 100kHz we can keep roughly the same number of nodes that we have in the FEE alcove.

Issues:

  • which fibers should be MPO vs LC?
    • only use MPO for fim's.
  • should we fanout timing at SRCF or put it through the bos?
    • use the fanouts at SRCF for simplicity
    • should we fanout in 208?
    • maybe put camlink data fibers through the bos?
  • routing mono-encoder udp network over to srcf
  • sharing xtcav between hutches with bos-switchable timing

100kHz data volumes:

  • wave8: 8waveforms*256samples/sample*2bytes/sample*0.1MHz = 400MB/s (need to add FEX data too)
  • piranha: 2048px*2bytes/px*0.1MHz=410MB/s
  • hsd full wf: 60000samples*2bytes/sample*0.1MHz = 12GB/s
    • max waveform length appears to be ~60000 samples looking at xtc2 file?
    • FEX needs to reduce it to 4GB/s (3x)
  • bld: 336bytes/event*0.1MHz = 33MB/s (from xtcreader -f /cds/data/psdm/prj/public01/xtc/tmoc00118-r0222-s008-c000.xtc2)
  • timing: 309bytes/event*0.1MHz = 31MB/s

summary:

  • hook it up similarly to the way it is done in the FEE alcove, but more neatly
  • fanout the timing in 208 and put everything through the BOS

recabling proposal:

  • tmo timing (including pvadet, bld) (LC, SFP to QSFP)
  • rix timing (LC, SFP to QSFP)
  • (4) tmo camlink nodes (LC, needs timing)
  • (2) rix camlink nodes (LC, needs timing)
  • tmo/rix fim node (5 fims) (MPO, PLR4)
  • (2) rix hsd nodes (2 LC per node, LR4)
  • (7) tmo hsd nodes (2 LC per node, LR4)
  • peppex hsd node (LC LR4)
  • (2) tmo/rix ami
  • (2) tmo/rix teb
  • (2) tmo/rix mebuser
  • xtcav (LC, needs timing)
  • mono-encoder (LC, needs timing)
  • in SRCF:
    • let's try MPO for everything (since we only have 48 LC)
    • tdet (12, timing): SFP fiber (from an MPO breakout cable) SFP to QSFP converter
      • 3 MPO breakout, 12 SFP to QSFP converters, 12 SFP transceivers
    • ilv (10, hsd): SFP fiber with LR4 transceiver (no MPO)
      • 5 MPO breakout, 20 LR4
    • wave8 (1, XilinxKcu1500Pgp4_6Gbps_ddr): MPO fiber with PLR4 transceiver
      • 1 MPO, 1 PLR4
    • camlink (4, ClinkKcu1500Pgp2b): data (transceiver 0, up to 4 lanes/cameras) MPO fiber with PLR4, timing (transceiver 1), try MPO fiber with PLR4 (neater, but wastes fibers)
      • 8 MPO, 8 PLR4

current inventory (sept. 13, 2022):

  • 27 nodes needed
  • shutdown 30
  • 3 broken (cmp006, 13, 15)

SRCF cabling

drp-srcf-cmpNNNKCU firmwareComment
001DrpTDet
002DrpTDet
003DrpTDet
004XilinxKcu1500Pgp4_6Gbps_ddrQSFP 1 cage is not working
005DrpPgpIlv
006ClinkKcu1500Pgp2b
007ClinkKcu1500Pgp2b
008DrpTDet
009DrpPgpIlv
010DrpTDet
011ClinkKcu1500Pgp2b
012ClinkKcu1500Pgp2b
013
No KCU
014Lcls2EpixHrXilinxKcu1500Pgp4_6Gbps
015
No mlx5; Bad Ethernet
016DrpTDet
017DrpPgpIlv
018DrpPgpIlv
019DrpPgpIlv
020DrpPgpIlv
021DrpPgpIlv
022DrpPgpIlv
023DrpPgpIlv
024DrpPgpIlv
025DrpTDet
026DrpTDet
027DrpTDet
028DrpTDet
029DrpTDet
030DrpTDet


Cabling

Top Patch Panel:





cmp012 (timing) (close to the USB port)cmp012 (data) (away from USB port)

cmp006 (timing) (close to the USB port)

cmp006 (data) (away from USB port)

1: cmp022(away from USB port)

2: cmp022(close to USB port)

3: cmp023 (away from USB port)

4: cmp023 (close to USB port)

1: cmp018(away from USB port)

2: cmp018(close to USB port)

3: cmp019 (away from USB port)

4: cmp019 (close to USB port)

1: cmp009(away from USB port)

2: cmp009(close to USB port)

3: cmp017 (away from USB port)

4: cmp017 (close to USB port)

1: cmp008

2: cmp010

3: cmp016

4: cmp025







cmp011 (timing) (close to the USB port)cmp011 (data) (away from USB port)

1: cmp024(away from USB port)

2: cmp024(close to USB port)

3: -

4: -

1: cmp020(away from USB port)

2: cmp020(close to USB port)

3: cmp021 (away from USB port)

4: cmp021 (close to USB port)

1: cmp029

2: cmp030

3: -

4: -

1: cmp026

2: cmp027

3: cmp028

4: -













TMO Hutch Fiber Connections

SourceDestinationNotes
XPM 2 input on RTMRack 4 in TMO hutch, FODU2, Cassette 4, fibers 3-4 (a.k.a TMO2.4.3-4)Note that cassettes 3 and 4 are swapped between rack 3/4
XPM 2 network connectionRack 3, Cassette 3, fibers 23-24, goes to Rack 4 FODU 5, Cassette 4, fibers 23-24, goes to Rack 4 FODU 2 cassette 4 fibers 5-6Note that cassettes 3 and 4 are swapped between rack 3/4

RIX Hutch Fiber Connections

SourceDestinationNotes
XPM 3 network connectionRack 1 FODU Cassette 1 fibers 15-16Don't immediately know what the official fodu number is

Room 208 Fiber Connections

See FEE Alcove Fiber Cabling for description of which patch panel in 208 goes to which detector.  Note that the fiber numbering on that page is different from here.  On the FEE alcove page we number fibers left-to-right, then top-to-bottom.  Here we use a better scheme: using the actual fiber numbers marked on the cassettes.  We should convert the FEE-alcove page to use the better fiber-numbering scheme.

SourceDestinationNotes
room 208 ATCA crate switch optical fiber port 0SRCF LC cassette 4 (counting from 1, left to right) fibers 11/1210Gb network to SRCF node (drp-srcf-cmp004?)
TMO XPM 2 network connection (TMO FODU 1, Cassette 4 fibers 5-6)ATCA switch optical port 7 counting from 0needed to swap tx/rx fibers to get link-lock
RIX XPM 3 network connection (RIX FODU, cassette 1, fibers 15-16)ATCA switch optical port 6 counting from 0
Timing output of XPM2 in 208TMO FODU 1, Cassette 3, Fibers 17-18: 1.3.17-18Cassette 1 is missing
Timing output of XPM3 in 208RIX FODU 1, Cassette 1, Fibers 7-8: 1.1.7-8
XPM0-AMC0-PORT0RIX 1.1.3-4This is XPM0 → XPM3 in RIX, FODU 1 for RIX is FODU 5 in 208
XPM0-AMC0-PORT1TMO 2.4.3-4This is XPM0 → XPM2 in TMO, FODU 2 for TMO is FODU 7 in 208
XPM5-AMC0-PORT0RIX 1.1.3-4This is XPM3→ XPM5
XPM6-AMC0-PORT0TMO 1.3.17-18This is XPM2→ XPM6
TMO ATM CAMLINKTMO FODU 1 cassette 3 fiber 1-2 (a.k.a. TMO1.3.1-2)cassette 1 is missing, swapped tx/rx at bos
RIX FIM MR4K2RIX FODU 1 cassette 1 fibers 25-26 (a.k.a RIX1.1.25-26)
RIX FIM MR3K2RIX1.1.27-28
RIX FIM CHEMRIXRIX1.1.29-30
TMO FIM1TMO1.3.5-6Not sure which FIM this really corresponds to in TMO
TMO FIM2TMO1.3.7-8Not sure which FIM this really corresponds to in TMO
RIX HSD2RIX1.1.19-20I may be swapping hsd2/3 (this was FEE fodu 1.3.18, old numbering scheme)
RIX HSD3RIX1.1.23-24I may be swapping hsd2/3 (this was FEE fodu 1.3.15, old numbering scheme)
TMO CAMLINK OPAL1TMO1.4.1-2swapped tx/rx at bos
TMO CAMLINK OPAL2TMO1.3.3-4swapped tx/rx at bos
TMO CAMLINK FZPTMO1.2.11-12swapped tx/rx at bos
RIX CAMLINK ATMRIX1.3.13-14
TMO HSD3 or companionTMO1.4.13-14I may be swapping hsd3 with its companion (don't know the name: not in tmo.cnf?).  This one was in fee fodu1.1.11 (old numbering scheme).


XPM Topology

globalTiming -> newxpm0 -> 2 (tmo) -> 4 (tmo)
                                   -> 6 (208) -> BOS
                        -> 3 (rix) -> 5 (208) -> BOS






  • No labels