with claus, weaver, cpo
Matt's proposal:
this is an argument for "specialized" cabling on the SRCF side:
3 sets of limited fiber resources:
(singlemode plr4 qsfp)
25
Cisco QSFP-40GE-LR4 Compatible 40GBASE-LR4 QSFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (single mode lr4)
45
Cisco CVR-QSFP-SFP10G Compatible 40G QSFP+ to 10G SFP+ Adapter Converter Module (sfp to qsfp adapter) CVR-QSFP-SFP10G
60
Cisco SFP-10G-LR Compatible 10GBASE-LR SFP+ 1310nm 10km DOM Duplex LC SMF Optical Transceiver Module (singlemode sfp transceiver (for sfp-to-qsfp adapters))
25
3m (10ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (3m singlemode breakout)
20
1m (3ft) MTP® Female to 4 LC UPC Duplex 8 Fibers Type B Plenum (OFNP) OS2 9/125 Single Mode Elite Breakout Cable, Yellow (1m single mode breakout)
45
Customized 8-144 Fibers Senko MPO-12 OS2 Single Mode Elite Trunk Cable, Yellow (3m mpo12 trunk cable)
20
0.5m (1.6ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (0.5m patch fiber)
20
1m (3ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1m patch fiber)
20
1.5m (5ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (1.5m patch fiber)
20
2m (7ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (2m patch fiber)
20
3m (10ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (3m patch fiber)
10
4m (13ft) LC UPC to LC UPC Duplex OS2 Single Mode PVC (OFNR) 2.0mm Fiber Optic Patch Cable (4m patch fiber)
Executive summary: I think at 100kHz we can keep roughly the same number of nodes that we have in the FEE alcove.
Issues:
100kHz data volumes:
summary:
recabling proposal:
current inventory (sept. 13, 2022):
drp-srcf-cmpNNN | KCU firmware | Comment | Application (Oct 17, 2022) |
---|---|---|---|
001 | DrpTDet | TMO low rate, XTCAV nic | slow tmo tdet (+xtcav) |
002 | DrpTDet | RIX low rate, encoder nic | slow rix tdet (+mono) |
003 | DrpTDet | TMO high rate | tmo high rate tdet |
004 | XilinxKcu1500Pgp4_6Gbps_ddr | rix/tmo wave8 Collection host | |
005 | DrpPgpIlv | rix hsd's | |
006 | ClinkKcu1500Pgp2b | LCLSECSD-1844 | nothing |
007 | ClinkKcu1500Pgp2b | tmo fzp piranha | |
008 | DrpTDet | didn't show up on xpm | rix user meb |
009 | DrpPgpIlv | rix hsd's | |
010 | DrpTDet | RIX high rate | rix fast tdet |
011 | ClinkKcu1500Pgp2b | tmo fzp opal | |
012 | ClinkKcu1500Pgp2b | ||
013 | ClinkKcu1500Pgp2b | tmo atm piranha | |
014 | Lcls2EpixHrXilinxKcu1500Pgp4_6Gbps | epix hr | |
015 | No mlx5; Bad Ethernet LCLSECSD-1305 Unreliable machine: bad for TEB / MEB | nothing | |
016 | DrpTDet | No mlx5. Error messages LCLSECSD-1305 Unreliable machine: bad for TEB / MEB | nothing |
017 | DrpPgpIlv | tmo hsd_3 | |
018 | DrpPgpIlv | tmo hsd_5,hsd_7 | |
019 | DrpPgpIlv | tmo hsd_4,hsd_13 | |
020 | DrpPgpIlv | tmo hsd_2 | |
021 | DrpPgpIlv | tmo hsd_6 | |
022 | DrpPgpIlv | tmo hsd_8,hsd_9 | |
023 | DrpPgpIlv | peppex hsd's | |
024 | DrpPgpIlv | tmo hsd_10,hsd_11 | |
025 | DrpTDet | rix slow tdet | |
026 | ClinkKcu1500Pgp2b | tmo atm opal | |
027 | ClinkKcu1500Pgp2b | rix atm opal | |
028 | DrpTDet | XPM:6 | tmo ami |
029 | DrpTDet | XPM:5 | rix ami |
030 | DrpTDet | NC | tmo user meb |
031 | XilinxKcu1500Pgp4_6Gbps_ddr | LCLSECSD-2030: BIOS not upgradable | tmo user meb |
032 | DrpTDet | LCLS1 HXR hutches | |
033 | DrpTDet | XPM:0. update bios settings: only one pcie bus. | |
034 | DrpTDet | mona slurm tests | |
035 | DrpTDet | XPM:0; LCLSECSD-2032: BIOS not upgradable from v2.0 but is useable as-is | mona slurm tests |
036 | DrpTDet | XPM:7 | epixM |
037 | DrpTDet | ||
038 | DrpTDet | XPM:0 | |
039 | DrpTDet | LCLSECSD-2028: Broken IPMI | |
040 | DrpTDet | XPM:0 | |
041 | DrpTDet | XPM:0 | |
042 | DrpTDet | XPM:0 | |
043 | DrpTDet | XPM:0 | EpixHrEmu |
044 | DrpTDet | XPM:0 | Laser hall |
045 | Todo: XilinxKcu1500Pgp4_6Gbps | XPM:0 | TXI Wave8 |
mon001 | tmo meb and ami_manager neh-base.cnf, hsd.cnf, rix-hsd.cnf | ||
046 | DrpPgpIlv | TMO HSD_15 and TMO HSD_16 | |
048 | DrpPgpIlv | TMO HSD_17 and TMO HSD_18 | |
049 | DrpPgpIlv | not working, possibly defective KCU | |
050 | DrpPgpIlv | TMO HSD_01 and TMO HSD_19 |
drp-srcf-gpuNNN | KCU firmware | Comment | Application |
---|---|---|---|
001 | in progress | Adding kcu1500 | |
002 | n/a | ||
003 | n/a | ||
004 | n/a |
NOTE: the rightmost MPO connections are MPO12 (2 rows, 4 columns). The left ones are MPO8. We should use MPO8 for KCU. The MPO12 are reserved for "future expansion" (can be converted to MPO8, I think).
(6-strand breakout cable) | DON'USE FOR NOW | DON'T USE FOR NOW | (breakout cable) 1: cmp022(away from USB port) 2: cmp022(close to USB port) 3: cmp023 (away from USB port) 4: cmp023 (close to USB port) | (breakout cable) 1: cmp018(away from USB port) 2: cmp018(close to USB port) 3: cmp019 (away from USB port) 4: cmp019 (close to USB port) | (breakout cable) 1: cmp009(away from USB port) 2: cmp009(close to USB port) 3: cmp017 (away from USB port) 4: cmp017 (close to USB port) | (breakout cable) 1: cmp008 2: cmp010 3: cmp016 4: cmp025 | |||||
(break cable) 1: cmp033 (away from usb port) 2: cmp035 (away from usb port) 3: cmp036 (away from usb port) 4: cmp037 (away from usb port) | (breakout cable) 2: cmp046 (away from usb) | (breakout cable) 1: cmp024(away from USB port) 2: cmp024(close to USB port) 3: - 4: - | (breakout cable) 1: cmp020(away from USB port) 2: cmp020(close to USB port) 3: cmp021 (away from USB port) 4: cmp021 (close to USB port) | (breakout cable) 1: cmp029 2: cmp030 3: cmp032 4: - | (breakout cable) 1: cmp028 2: - 3: - 4: - | ||||||
(breakout cable) |
cmp006 (timing) (close to the USB port) | cmp011 (timing) (close to the USB port) | cmp012 (timing) (close to the USB port) | cmp004(close to usb port) | cmp013(timing) (close to the USB port) | cmp026(timing) (close to the USB port) | cmp027(timing) (close to the USB port) | cmp004(away from usb port) |
cmp006 (data) (away from USB port) | cmp011 (data) (away from USB port) | cmp012 (data) (away from USB port) | cmp013 (data) (away from USB port) | cmp045 (away from USB port) | cmp026 (data) (away from USB port) | cmp027 (data) (away from USB port) | cmp014(away from usb port) |
IT | IT | cmp007 (close to the USB port) | cmp001 (away from the USB port) | |||
IT | cmp007 (away from the USB port) | cmp002 (away from the USB port) | ||||
IT | cmp003 (away from the USB port) | |||||
IT | cmp005 (close to the USB port) | |||||
IT | cmp005 (awy from the USB port) | |||||
IT |
Fibers 15-16: XTCAV from Room 208 Rack 2 PH2 fibers 61-62 → cmp001
Fibers 21-22: MonoEncoder from ACTA Switch in FEE → cmp002
Source | Destination | Notes |
XPM 2 input on RTM | Rack 4 in TMO hutch, FODU2, Cassette 4, fibers 3-4 (a.k.a TMO2.4.3-4) | Note that cassettes 3 and 4 are swapped between rack 3/4 |
XPM 2 network connection | Rack 3, Cassette 3, fibers 23-24, goes to Rack 4 FODU 5, Cassette 4, fibers 23-24, goes to Rack 4 FODU 2 cassette 4 fibers 5-6 | Note that cassettes 3 and 4 are swapped between rack 3/4 |
Source | Destination | Notes |
---|---|---|
XPM 3 network connection | Rack 1 FODU Cassette 1 fibers 15-16 | Don't immediately know what the official fodu number is |
XPM 3 AMC1 port 5 counting from 0 | Room 208 RIX1 fodu, cassette 1 fibers 11-12 | Wasn't connected on the room 208 side on Oct. 26, 2022. Maybe not necessary any more? |
Source | Destination | Notes |
---|---|---|
BOS 1.8.1, 1.8.2 and XPM7 output | Building B950 RB02 in room 208: SM S04 To B999 MEZZ FODU 43 R11, Ports 1,2,4 | 3 fiber pairs for epixHR data (two pairs) and timing (one pair) |
See FEE Alcove for description of which patch panel in 208 goes to which detector. Note that the fiber numbering on that page is different from here. On the FEE alcove page we number fibers left-to-right, then top-to-bottom. Here we use a better scheme: using the actual fiber numbers marked on the cassettes. We should convert the FEE-alcove page to use the better fiber-numbering scheme.
Source | Destination | Notes |
---|---|---|
ATCA FEE router port 0 | ATCA room 208 router port 1 | 10Gb network to SRCF node, the connection is done via the octopus fiber #4 |
room 208 ATCA crate switch optical fiber port 0 | SRCF LC cassette 4 (counting from 1, left to right) fibers 11/12 | 10Gb network to SRCF node (drp-srcf-cmp004?) |
TMO XPM 2 network connection (TMO FODU 1, Cassette 4 fibers 5-6) | ATCA switch optical port 7 counting from 0 | needed to swap tx/rx fibers to get link-lock |
RIX XPM 3 network connection (RIX FODU, cassette 1, fibers 15-16) | ATCA switch optical port 6 counting from 0 | |
Mono Encoder Network | From FEE Fodu 1 cassette 3 fibers 1,2 to SRCF LC fodu fibers 21,22 | |
Timing output of XPM2 in 208 | TMO FODU 1, Cassette 3, Fibers 17-18: 1.3.17-18 | Cassette 1 is missing |
Timing output of XPM3 in 208 | RIX FODU 1, Cassette 1, Fibers 7-8: 1.1.7-8 | |
XPM0-AMC0-PORT0 | RIX 1.1.3-4 | This is XPM0 → XPM3 in RIX, FODU 1 for RIX is FODU 5 in 208 |
XPM0-AMC0-PORT1 | TMO 2.4.3-4 | This is XPM0 → XPM2 in TMO, FODU 2 for TMO is FODU 7 in 208 |
XPM0-AMC1-PORT2 | TMO 2.3.33-34 (tmo cassette 3 fiber 33-34) | Backup connection for XPM0->XPM2 |
XPM5-AMC0-PORT0 | RIX 1.1.3-4 | This is XPM3→ XPM5 |
XPM6-AMC0-PORT0 | TMO 1.3.17-18 | This is XPM2→ XPM6 |
XPM0-AMC1-PORT0 | FEE FODU 1, Cassette 4 fiber 11-12 (top right) | LCLS2 timing for FEE Teststand |
TMO ATM CAMLINK | TMO FODU 1 cassette 3 fiber 1-2 (a.k.a. TMO1.3.1-2) | cassette 1 is missing, swapped tx/rx at bos |
RIX FIM MR4K2 | RIX FODU 1 cassette 1 fibers 25-26 (a.k.a RIX1.1.25-26) | |
RIX FIM MR3K2 | RIX1.1.27-28 | |
RIX FIM CHEMRIX | RIX1.1.29-30 | |
TMO FIM1 | TMO1.3.5-6 | Not sure which FIM this really corresponds to in TMO |
TMO FIM2 | TMO1.3.7-8 | Not sure which FIM this really corresponds to in TMO |
RIX HSD2 | RIX1.1.19-20 | I may be swapping hsd2/3 (this was FEE fodu 1.3.18, old numbering scheme) |
RIX HSD3 | RIX1.1.23-24 | I may be swapping hsd2/3 (this was FEE fodu 1.3.15, old numbering scheme) |
TMO CAMLINK OPAL1 | TMO1.4.1-2 | swapped tx/rx at bos |
TMO CAMLINK OPAL2 | TMO1.3.3-4 | swapped tx/rx at bos |
TMO CAMLINK FZP | TMO1.2.11-12 | swapped tx/rx at bos |
RIX CAMLINK ATM | RIX1.3.13-14 | |
TMO HSD3 or companion | TMO1.4.13-14 | I may be swapping hsd3 with its companion (don't know the name: not in tmo.cnf?). This one was in fee fodu1.1.11 (old numbering scheme). Needed tx/rx swap at bos. |
TMO HSD3 or companion | TMO1.4.15-16 | I may be swapping hsd3 with its companion (don't know the name: not in tmo.cnf?). This one was in fee fodu1.1.9 (old numbering scheme). Needed tx/rx swap at bos. |
TMO HSD5 or HSD7 | TMO1.4.31-32 | I may be swapping hsd5/hsd7. This one was in fee fodu 1.1.1 (old numbering scheme). Needed tx/rx swap at bos. |
TMO HSD5 or HSD7 | TMO1.4.29-30 | I may be swapping hsd5/hsd7. This one was in fee fodu 1.1.5 (old numbering scheme). Needed tx/rx swap at bos. |
TMO HSD4 | TMO1.4.27-28 | Was in fee fodu 1.1.4 (old numbering scheme). tx/rx swap at bos. |
TMO HSD4 companion | TMO1.4.33-34 | Was in fee fodu 1.1.6 (old numbering scheme). tx/rx swap at bos. |
TMO HSD1 | TMO1.4.17-18 | Was in fee fodu 1.1.12 (old numbering scheme). tx/rx swap at bos. |
TMO HSD2 | TMO1.4.19-20 | Was in fee fodu 1.1.8 (old numbering scheme). NO tx/rx swap at bos! |
TMO HSD6 | TMO1.4.35-36 | Was in fee fodu 1.1.3 (old numbering scheme). tx/rx swap at bos. |
TMO HSD6 companion | TMO1.4.25-26 | Was in fee fodu 1.1.2 (old numbering scheme). tx/rx swap at bos. |
TMO HSD9 | TMO1.4.21-22 | Was in fee fodu 1.1.7 (old numbering scheme). tx/rx swap at bos. |
TMO HSD8 | TMO1.4.23-24 | Was in fee fodu 1.1.10 (old numbering scheme). tx/rx swap at bos. |
RIX HSD1 | RIX1.1.21-22 | Was in fee fodu 1.3.8 (old numbering scheme) |
RIX HSD0 | RIX1.1.17-18 | Was in fee fodu 1.3.9 (old numbering scheme) |
PEPPEX HSD1 | Room 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 21-22 (second from bottom). Also labelled "slot 4" on the slot itself. | Was in fee fodu 2.4.3 (old numbering scheme) |
PEPPEX HSD0 | Room 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 19-20 (third from bottom). Also labelled "slot 4" on the slot itself. | Was in fee fodu 2.4.4 (old numbering scheme) |
PEPPEX OPAL | Room 208 Rack 2 PH4 "To B920 R90-S SLOT B" fibers 23-24 (bottom pair). Also labelled "slot 4" on the slot itself. | Was in fee fodu 2.4.5 (old numbering scheme) |
XTCAV | Room 208 Rack 2 PH2 fibers 61-62 from B005 (each cassette has 12 fibers, so top of sixth cassette from left) | Was in fee fodu 1.2.2 (old numbering scheme) |
TXI FIM | Room 208
Data To FODU 8 Cassete 2 Port 11-12 | |
TMO LCLS1 timing | XPM7 AMC1 port1 to TMO FODU 7 Cassette 3 (counting from 1 with 4 cassettes total, one currently unoccupied) fibers 11-12 | |
XPM0-AMC1-PORT4 | FEE FODU 6, Cassette 3 fiber 23-24 (lower right) | LCLS2 timing for FEE Teststand XPM10 RTM |
TMO HSD 88 or 89 | BOS (2.1)1-2 or 5-6 | In TMO Hutch connected to FODU1 Cassette 1 position 8 (most left in the middle row might be broken) and 15 (lowest row int he middle) RACK 4 in 208 FODU 7 RACK 4 |
TMO HSD 89 or 88 | BOS(2.1) 5-6 or 1-2 | in TMO Hutch connected to FODU1 Cassette 1 position 9 and 10 RACK 4 in 208 FODU 7 RACK 4 N.B. position 9 (in the middle row third from the left) was already taken in 208 but not in TMO. Connected to FODU4 (208) cassette 2 (second from the left) position 7 (first from the left in the middle), but removed because no fiber in TMO. |
TMO HSD 1 (1B:B) | BOS 1.8.3 | In TMO hutch connected to FODU1 Cassette 1 position 3 (counting from top left from 1) RACK4 |
TMO HSD 18 (B1:B) | BOS 1.8.4 | In TMO hutch connected to FODU1 Cassette 1 position 4 (counting from top left from 1) RACK4 |
RIX High-rate encoder in FEE (data and timing, in that order) | From FEE: B940-008-R03-FODU-U1 PAIRS 2 and 3 COUNTING FROM 1 Through FEE alcove: B940-009-R06-FOD3-U1 pairs 2 and 5 counting from 1 To B950-208-r42-FOD3-U2 PAIRS 2 and 5 COUNTING FROM 1 | Swapped tx/rx at xpm5 amc1 port 2 (counting from 0). Data also needed to be swapped going into BOS (1.7.8). |
XPM Topology
lcls2Timing -> xpm0 -> 2 (tmo) -> 4 (tmo) -> 6 (208) -> BOS -> 3 (rix) -> 5 (208) -> BOS -> 10(fee) -> 11(fee) lcls1Timing -> xpm7 |
Cleanup: