To do list:

• Add the ZYNQ based option

• Assume "IF" we can do compression, then factor of 3 or better compression factor
• Use the same FPGA that's used in LCLS-II to get a big discount
• Assumes channel aggregation at the WIB
  • 320 channels per link @ 10 Gbps (64/66 encoding)
  • 2 links per PCIe (no compression)
  • 4 links per PCIe (with compression)
• If "passive WIB" option (1Gbps/link), then replace the QSFPs with SNAP12 from reflexphotonics
  • Use the SELECTIOs instead of GTs in this slow rate condition
• This FPGA has 20 GT channels
  • 1x lane for FPGA based 10 GbE
  • 3x lanes for SLI interface
  • 4x lanes for QSFP
  • 8x lane for PC's PCIE interface
  • 4x lanes for M.2 PCIE interface
• Support up to 3x PCIe interfaces (only 2 required)
• Assuming timing network only requires LVDS I/O (same as proto-DUNE)
• Number of PCIe card per APA
  • 4x PCIeCard per APA (no compression)
  • 2x PCIeCard per APA (with compression)
• Use a SLI interface to create a full mesh point-to-network for 1 APA
  • Example of cable
  • Example of SLI interface
  • 4-way SLI cable for no compression
  • 2-way SLI cable for with compression
• Can support up to 12x PCIe slots per server
  • Quote for $5.25k/server 40 unit production
  • Assume 300W for everything in the server (expect for the SLAC PCIe cards)
  • 2 APA per rack server server without compression
  • 4 APA per rack server server with compression
  • 4u per rack server
  • Full Specs: https://www.supermicro.com/products/system/4u/4028/sys-4028gr-tr.cfm
  • Price based on http://www.serversdirect.com
• Reliability and Hot-Swapability
  
  • The power supplies and fans are the first things that we expect to fail
  • 8 Hot-Swap 92mm cooling fans
    • 4 x 2 configuration

  • 2000W Redundant (2+2) Power Supplies; Platinum Level (94%+)

• Backend communication through two 10GbE port (4028GR-TRT version)
• Standard 1GbE IPMI interface
• No fans on the FPGA for deployment servers
  
  • Cooling through the servers redundant fans
  • Up to 200 LFM airflow
  • Use ATS-P1-152-C1-R0 on the FPGA, which will have no air blockage from the QSFP/SFP
    

    • HEAT SINK 35MM X 35MM X 35MM

    • PCIe gap is is 40.64mm between slots (designed for double wide GPUs).

    • Picture of heatsink

  • Thermal Resistance @ Forced Air Flow: 3.29°C/W @ 100 LFM 

    • FPGA junction temperate would increase by ~50°C (15W x 3.29°C/W)

    • FPGA operational temperature: 0°C ~ 100°C (TJ)

    • This means max. operational ambient temperature would be 50°C

      • Expecting 25°C nominal ambient temperature

      • 25°C of design margin @ 100 LFM (up to 200 LFM on the fans)
• For development test stands that use a standard PC, a fan will be required
  
  • Because standard PC typically don't do force air on the PCIe slots
  • This PCIe card design will need to support both configurations as loading options
    
• Server's Operating Temperature: 10º to 35º C (50º to 95º F)
  • Recommend doing rack cooling if underground rack room temperature is not temperature controlled
  • The ATCA design will allow require racking cooling if if underground rack room temperature is not temperature controlled

Without Compression

• Power
  
  • Power per server: 630W = (300W/server) + (12 PCIeCard/server) x (27.5W/PCIeCard)
  • Power per system: 31.5kW = (50 server/system) x (520W/server)
• Rack Space
  • Space per server: 4u = (7" height) x (1u/1.75")
  • space per system: 200u = (50 server/system) x (4u/server)
• Cost
  • Cost per server: $32.922k = ($5.25/server) + (12 PCIeCard/server) x ($2.306/PCIeCard)
  • Cost per system: $1.65M = (50 servers/system) x ($32.922k)

Summary: