This is a specification document to help the firmware engineer to integrate the DaqMux with the application and the amc-carrier-core.

Overview

The DaqMux was originally developed as an oscilloscope to transmit streams from the FPGA to software for debugging purposes. Then it started being used as a data streamer by the application to software, and it's configurations were exported to EPICS/software. 

Feature summary

The DaqMux can perform the following functionalities

  • Multiplex up to 16 streams (lanes) and send forward (to software) - default is 4
  • Stream packets can be acquired continuously every time a trigger arrives in continuous mode, or only once upon trigger with or without header (meta data) in trigger mode, or many times with or without header (meta data) in trigger mode when auto re-arm is enabled
  • Possibility of cascading trigger of DaqMux blocks in case streams from several DaqMuxes acquired at the same instant is required
  • Pause streaming by not overwriting buffers (freeze functionality), therefore nothing is forwarded to software
  • Down sampling and averaging


Definitions

  • Stream : a group of bytes transmitted in sequence.
  • Lane : a physical bus at which the stream crosses 
  • Packet : User abstraction. Bunches of bytes that the user wants forward to software from each lane every time there is a trigger
  • AXI transaction : Hardware abstraction. Packets are broken into a 1 or more AXI transactions
  •  

Applications

Figure 1 shows the DaqMux instantiated in the Common platform firmware in the application side. For convenience, everything except for the AppCore is available and instantiated in the amc-carrier-core RTL. That allows the application firmware developer to concentrate on core of the application. Also, there is less margin for error.

Figure 1: Common platform firmware top-level

Generics & IO description

Table 1: DaqMux generics list and description

Generic namedefault valueDescription
TPD_G1Simulation variable; register clock to output delay in nanoseconds
DECIMATOR_EN_GTrueDecimator enable
WAVEFORM_TDATA_BYTES_G4Output lane width in bytes
FRAME_BWIDTH_G 10Power of two, which defines AXI stream transaction size in words (4 bytes)(i.e. when FRAME_BWIDTH_G = 12 → 210 x 4 = 4096 bytes)
BAY_INDEX_G-Index of the DaqMux
N_DATA_IN_G16Number of input data lanes (N_DATA_IN_G must be between 1 and 29)
N_DATA_OUT_G4Number of output data lanes (must be between 1 and 16)


Table 2: IO port list and description

CategoryNameDirectionClock domainWidthDescription




Clock & reset logic

axiClk Input-1AXI Lite clock
axiRst InputaxiClk1AXI Lite reset
devClk_i Input-1Development clock
devRst_i InputSynced to devClk_i in DaqMux (for some reason)1Development logic reset
wfClk_i Input-1Ouput lanes' clock
wfRst_i InputwfClk_i1Output lanes' reset









DaqMux control signals and timing information

trigHw_i InputSynced to devClk_i in DaqMux1

Trigger signal to start the DaqMux streaming

  • 0 indicates no trigger event
  • 1 indicates a trigger event
trigCasc_i InputSynced to devClk_i in DaqMux1

Cascaded trigger input. Can be used along with trigHw_i when enabled in the register file

  • 0 indicates no trigger event
  • 1 indicates a trigger event
trigCasc_o OutputdevClk_i1

Output trigger signal connected to the SW Trigger Enable control register

  • 0 indicates no trigger event
  • 1 indicates a trigger event
armCasc_i InputSynced to devClk_i in DaqMux1

Cascaded trigger Arm. Arms the trigger and prepares DaqMux for trigger arrival

  • 0 indicates no arm
  • 1 indicates armed
armCasc_o OutputdevClk_i1

Output cascade signal connected to the Arm HW Trigger control register

  • 0 indicates no arm
  • 1 indicates armed
freezeHw_i InputSynced to devClk_i in DaqMux1

Adds invalid flag to the streams that are forwarded, and they will be discarded in one of the posterior blocks in the pipeline

  • 0 indicates valid (not freeze)
  • 1 indicates invalid (freeze)
timeStamp_i InputSynced to devClk_i in DaqMux64Time stamp coming from the AMC carrier core
bsa_i InputSynced to devClk_i in DaqMux128BSA information coming from the AMC carrier core
dmod_i InputSynced to devClk_i in DaqMux192Dmod timing information coming from the AMC carrier core












AXI Lite register memory mapped interface for reading and writing to register file

axilReadMaster InputaxiClk1

AXI Lite record containing

Read Address channel

  • araddr  : slv(31 downto 0);
  • arprot  : slv(2 downto 0);
  • arvalid : sl;

Read data channel

  • rready  : sl;
axilReadSlave OutputaxiClk1

AXI Lite record containing

 Read Address channel

  • arready : sl;

Read data channel

  • rdata   : slv(31 downto 0);
  • rresp   : slv(1 downto 0);
  • rvalid  : sl;
axilWriteMaster InputaxiClk1

AXI Lite record containing

Write address channel

  • awaddr  : slv(31 downto 0);
  • awprot  : slv(2 downto 0);
  • awvalid : sl;

Write data channel

  • wdata   : slv(31 downto 0);
  • wstrb   : slv(3 downto 0);
  • wvalid  : sl;

Write ack channel

  • bready  : sl;
axilWriteSlave OutputaxiClk1

AXI Lite record containing

Write address channel

  • awready : sl;

Write data channel

  • wready  : sl;

Write ack channel

  • bresp   : slv(1 downto 0);
  • bvalid  : sl;


Input lane array - data valid based

sampleDataArr_i InputdevClk_iN_DATA_IN_GInput data stream array of 32-bit words
sampleValidVec_i InputdevClk_iN_DATA_IN_GInput valids with each valid corresponding to the respective stream in that cycle
linkReadyVec_i InputdevClk_iN_DATA_IN_G

Indicate that the stream source is ready. If is 0, the DaqMux sets the Stream Error control register to 1.







Output lane array

rxAxisMasterArr_o OutputwfClk_iN_DATA_OUT_G

AXI compliant output data stream array of 32-bit words. Nonetheless, the type of rxAxisMasterArr_o is defined as follows in the AxiStreamPkg.vhd file

  • tValid : sl;
  • tData  : slv(511 downto 0);
  • tStrb  : slv(63 downto 0);
  • tKeep  : slv(63 downto 0);
  • tLast  : sl;
  • tDest  : slv(7 downto 0);
  • tId    : slv(7 downto 0);
  • tUser  : slv(63 downto 0);
rxAxisSlaveArr_i InputwfClk_iN_DATA_OUT_GAXI streams data ready signals
rxAxisCtrlArr_iInputSynced to devClk_i in DaqMuxN_DATA_OUT_G

AXI stream control signals as follows

  • pause    : sl;
  • overflow : sl;
  • idle     : sl;

If pause is 1, the DaqMux will not start and error is set. Represents status(DONE_C) signal in the AxiStreamDmaRingWrite module of the waveforEngine. Not clear what is the meaning of this flag. Found this comment in that module

--  status(DONE_C) indicates a push, but maybe more than one



Register address mappings and description

Table 3: DaqMux register address mapping and description

Register name

Address

AccessBitsAliasDescription

Control

0x0RW

0

SW Trigger Enable

Triggers DAQ on all enabled channels. Must be set to 1, then set to 0 again.

1Cascade Trigger enable

Enabling/disabling cascaded trigger

  • '0' - Disable Cascaded Trigger
  • '1' - Enable Cascaded Trigger
2Auto Rearm HW Trigger

Enabling/disabling hardware automatic trigger. If disabled it has to be rearmed by Arm Hw Trigger

  • '0' - Disabled (has to be armed with bit3 otherwise disabled)
  • '1' - Enabled
3Arm HW TriggerArms the hardware trigger on rising edge. After trigger occurs the trigger has to be rearmed using this register.
4Trigger Clear StatusTrigger status will be cleared (On the rising edge).
5Daq Mode

Select the data acquisition mode ( Stream stops if Error occurs )

  • '0' - Trigger mode - Normal DAQ mode
    • Has to be triggered to start every time
  • '1' - Continuous mode - The data is framed and continuously streamed out after enabled. (Still requires a trigger to start)
    • Disable the stream to stop
6Packet Header Enable

Add 128-bit header (otherwise only data will be inserted)(Applies only to Triggered mode only)

  • '0' - Disabled
  • '1' - Enabled
7SW Freeze Buffer Freezes all enabled circular buffers
8HW Freeze Buffer Enable

Enabling/disabling hardware freeze buffer request

  • '0' - Disabled
  • '1' - Enabled
Status0x1RO

0

Software Trigger Status

Software Trigger Status (Registered on first trigger until cleared by Trigger Clear Status - Control[4] ).

1Cascade Trigger StatusCascade Trigger Status (Registered on first trigger until cleared by Trigger Clear Status - Control[4] )
2HW Trigger StatusHardware Trigger Status (Registered on first trigger until cleared by Trigger Clear Status - Control[4] )
3HW Trigger Armed StatusHardware Trigger Armed Status (Registered on rising edge Arm HW Trigger - Control[3] - and cleared when hardware trigger occurs )
4Combined Trigger StatusCombined Trigger Status (Registered when trigger condition is met until cleared by Trigger Clear Status - Control[4] )
5Freeze Buffers Status Freeze buffer occurred (Registered on first freeze until cleared by Trigger Clear Status - Control[4] )
Decimation0x2RW15:0Decimation Rate Divisor

Sample rate divider (Decimator):

  • Averaging Enabled: (powers of two) 1,2,4,8,16,etc (max 2^12)
  • Averaging Disabled: 1,2,3,4,etc (max 2^16-1)
DataSize0x3RW31:0Data Buffer Size

Number of 32-bit words to forward at each trigger (Ignored in continuous mode) (if enabled header will be included in the first 14 words of data). Minimum size is 14 (the size of the header).

TimeStamp0x4RO31:0Timestamp[31:0]Timestamp [31:0] - secPastEpoch
0x5RO31:0Timestamp[63:32]Timestamp [63:32] - nsec
BSA0x6RO31:0bsa(0)

edefAvgDn

0x731:0bsa(1)edefMinor
0x831:0bsa(2)edefMajor
0x931:0bsa(3)edefInit
TrigCount0xARO31:0Trigger CountCounts valid data acquisition triggers
DbgInputValid0xBRO31:0Debug Input ValidAll DaqMux AXI input streams valid signals 
DbgLinkReady0xCRO31:0Debug Link ReadyAll DaqMux AXI input streams ready signals
InputMuxSel0x10RW4:0Input Mux Select[0]

0x1x: Stream x: Channel select Multiplexer 

0 - Disabled, 1 - Test, 2 - Ch0, 3 - Ch1, 4 - Ch2 etc.(up to Ch29)

Test mode will output counter data

0x11-0x1E4:0Input Mux Select[1]-[14]
0x1F4:0Input Mux Select[15]
DaqStatus0x20-0x2F (one register for each of the 16 (enabled) lanes)RO0Stream Pause

(rxAxisCtrlArr_i port signal) Raw diagnostic stream control pause (Waveform engine bufferDone signal. When an AXI frame is completely  written to DRAM, this bit is set)

1Stream ReadyDebug flag: Raw diagnostic stream control Ready (Waveform engine FIFO output stream ready signal)
2Stream Overflow(rxAxisCtrlArr_i port signal) Raw diagnostic stream control Overflow (set to 0 in waveform engine)
3Stream ErrorDebug flag: Error during last Acquisition (Raw diagnostic stream control Ready or incoming data valid dropped)
4Data source readyDebug signal: The data source is ready. If not, DaqMux sets error signal to 1
5Stream EnableDebug signal: Output stream enabled
31:6Frame CountNumber of 4096 word frames sent
DataFormat


0x30-0x3F (one register for each of the 16 (enabled) lanes)


RW


4:0Sign Bit PositionIndicating sign extension point (all bits after sign bit will be overwritten with sign)
5Data Width

Data width 32-bit or 16-bit

  • '0' : 32-bits
  • '1' : 16-bits
6Sign enable

Signed/unsigned

  • '0' : Unsigned
  • '1' : Signed
7Decimation Averaging Enable

Decimation Averaging

  • '0' : Disable
  • '1' : Enable

Functional description & block diagrams

The DaqMux has two data acquisition (Daq) modes: Trigger mode and continuous mode. The names are deceiving because both modes need a trigger to operate. A more appropriate description may be automatic (continuous) and manual (trigger) modes. The DaqMux block diagram is shown in Figure 2.

Figure 2: DaqMux Block diagram


The block diagram is composed of the following blocks:

  • Input synchronization logic: Logic to synchronize input signals to devclk_i. Multi-bit signals are synchronized with double flopping as well as it is assumed that they will be stable 1/360s in worst case scenario
  • DaqTrigger: Receives all trigger sources and generates a single-cycle trigger and freeze signals to be used in the rest of the logic
  • DaqLane[X]: Instantiated N_DATA_OUT_G times. Each DaqLane receives all input streams performs multiplexing upon trigger, generates headers, decimation and averaging if enabled
  • Decimator & Avg: Performs the decimation (down-sampling) and averaging
  • FIFO[X]: Dual clock fifo to synchronize output lane array to wfClk_i
  • DaqRegltf: AXI Lite memory mapped register file that contains control and status registers of the DaqMux

A state machine diagram of the DaqMux operation is shown in Figure 3. For more detailed explanations, see the following sections.


Figure 3: DaqMux operation state machine

Continuous mode

As mentioned earlier, the continuous mode multiplexes raw data, and outputs either the raw data or the down sampling with or without averaging. It is activated with a trigger, and terminated upon error or lane disabling. After termination a new trigger is required. detailed state transition is described as follows:

Idle state exit conditions

The continuous mode starts in the presence of all of the following conditions

  • rxAxisCtrl input pause signal is set to 0 (Comes from the waveform engine of the amc-carrier-core) Not documented anywhere, but a speculation is when the buffers in the end of the pipeline are full, the DaqMux stops forwarding data. A much less confusing solution and implementation would be to use the back pressure feature of AXI streaming (ready goes to 0).
  • LinkReadyVec_i is 1. This indicates that the Jesd is ready
  • Control bit 5: this bit indicates that continuous mode is chosen

And one of the following conditions:

  • Software trigger indicated by control bit 0
  • Cascaded trigger indicated by both the setting of the cascaded arm and trigger signals
  • Hardware trigger indicated by the input trigger signal trigHw_i and the arm or auto arm control registers

Stream data state

In the stream data state one of the following is transmitted from the multiplexed lanes

  • Raw data
  • Down sampled data with or without averaging

Stream data termination conditions

In continuous mode, the stream state is exited in the following conditions

  • An error is established
  • The channel is disabled

Trigger mode

The trigger mode multiplexes raw data, and outputs a burst number of words upon trigger arrival. The bust may or may not contain a header (not available in continuous mode), and the burst data can be either the raw data or the down sampling with or without averaging. It is activated with a trigger, and terminated upon error . After termination a new trigger is required. detailed state transition is described as follows:

Idle state exit conditions

The continuous mode starts in the presence of all of the following conditions

  • rxAxisCtrl input pause signal is set to 0 (Comes from the waveform engine of the amc-carrier-core) Not documented anywhere, but a speculation is when the buffers in the end of the pipeline are full, the DaqMux stops forwarding data. A much less confusing solution and implementation would be to use the back pressure feature of AXI streaming (ready goes to 0).
  • LinkReadyVec_i is 1. This indicates that the Jesd is ready
  • Control bit 5: this bit indicates that trigger mode is chosen

And one of the following conditions:

  • Software trigger indicated by control bit 0
  • Cascaded trigger indicated by both the setting of the cascaded arm and trigger signals
  • Hardware trigger indicated by the input trigger signal trigHw_i and the arm or auto arm control registers

Stream data state

In the stream data state a burst of predefined number of the following is forwarded

  • A fixed number of words containing raw data words
  • A fixed number of words containing a header followed by raw data words
  • A fixed number of words containing header followed by down sampled data with or without averaging

Stream data termination conditions

The stream state is exited in the following conditions

  • An error is established
  • The channel is disabled
  • Request packet size is reached

The header data is 14 32-bit words as follows:

  • HeaderWord[0]: dmod(31:0) - Timing pattern information
  • HeaderWord[1]: dmod(63:32) - Timing pattern information 
  • HeaderWord[2]: dmod(95:64) - Timing pattern information 
  • HeaderWord[3]: dmod(127:96) - Timing pattern information 
  • HeaderWord[4]: dmod(159:128) - Timing pattern information 
  • HeaderWord[5]: dmod(191:160) - Timing pattern information 
  • HeaderWord[6]: timeStamp_i(31:0)   –  secPastEpoch
  • HeaderWord[7]: timeStamp_i(63:32)  –  nses 
  • HeaderWord[8]: bsa(127:96) - edefAvgDn
  • HeaderWord[9]: bsa(95:64) -edefMinor
  • HeaderWord[10]: bsa(63:32) -edefMajor
  • HeaderWord[11]: bsa(31:0) -edefInit
  • HeaderWord[12]: packetSize_i
  • HeaderWord[13]: Firmware specific signals
    • [BIT31:BIT27] = "00000"
    • [BIT26:BIT26] = Trigger Header: Hardware Trigger – s_trigHw
    • [BIT25:BIT25] = Trigger Header: Cascaded Trigger – s_trigCascRe
    • [BIT24:BIT24] = Trigger Header: Software Trigger – s_trigSw
    • [BIT23:BIT23] = Data format – dec16or32_i
    • [BIT22:BIT22] = Enable decimation averaging – averaging_i
    • [BIT21:BIT21] = Test mode – test_i
    • [BIT20:BIT20] = AMC BAY Index – BAY_INDEX_G
    • [BIT19:BIT16] = Channel Index – axiNum_i
    • [BIT15:BIT00] = decimation rate divide – rateDiv_i

Clocking/performance requirements

The maximum clock frequency has not been evaluated, nonetheless, this logic was used with clock frequencies as follows

  • devClk_i : 185 MHz
  • wfClk_i : (set to axiClk in the amc-carrier-core) 156.25 MHz
  • axiClk : 156.25 MHz

Connectivity

As mentioned earlier, the AppTop which contains the DaqMux is already instantiated. The firmware application developer will need to develop the APPCore. The following (just informative) DaqMux ports are routed to the waveform engine in the amc-carrier-core, where the rest of the magic moving this data to the DRAM and from the DRAM to the CPU memory happens.

  • wfClk_i
  • wfRst_i
  • rxAxisMasterArr_o
  • rxAxisSlaveArr_i
  • rxAxisCtrlArr_i

Signals timngStamp_i , bsa_i, and dmod_i come from the timingTrig record that is also provided by the amc-carrier core.

The signals that the application firmware developer needs to worry about are as follows:

  • freezeHw_i
  • trigHw_i
  • DAC input signals
    • Data - sampleDataArr_i[13:7] : AppCore dacValues[6:0]
    • Valids - sampleValidVec_i[13:7]: AppCore dacValids[6:0]
  • Debug input signals:
    • Data - sampleDataArr_i[17:14]: AppCore debugValues[3:0]
    • Valid - sampleValidVec_i[17:14] : AppCore debugValids[3:0]

  • LinkReady signals: Mapped to valids of all stream except for debug

    • LinkReady[17:14]: 4'hF
    • LinkReady[13:7]: dacValids[6:0]
    • LinkReady[6:0]: adcValids[6:0]



  • No labels