Page History
Current Development Machine Name
lcls-pc83236 # machine with the kcu1500 in 901
lcls-pc88284 # Silke's machine
drp-tst-acc02 # machine with the kcu1500 in the setup lab
rdsrv223 # machine with usb connection to kcu1500 for jtag flashing and integrated logic analyzer (ila) functionality
Preparing git ssh keys
Add your ssh keys to git (This is unfortunately necessary because the .gitmodules file in lcls2-pcie-apps uses the "git" form of the URL instead of the "http" form):
...
Also note that the firmware requires a new version of git that supports "links to large files" (lfs). Add /afs/slac/g/reseng/git/git/bin to PATH.
Rogue Documentation
https://slaclab.github.io/rogue/
Conda Commands to Create Rogue Environment
This is in addition to the other packages that must be built for the rogue library (see next step). These conda commands are derived from:
...
Code Block |
---|
conda env create -n timetool source activate timetool conda install pyyaml conda install pyzmq conda install -c conda-forge parse conda install click conda install MySQLdb conda install -c bioconda mysqlclient conda install -c conda-forge pyro4 conda install numpy pip install recordclass |
Building Rogue
git clone https://github.com/slaclab/rogue.git
...
Some applications are not built by default. cd to directory and make.
Building Firmware
Follow instructions in the README.md here (make sure to use the modern AFS version of git described here so you can use git-lfs):
...
cd firmware/targets/TimeToolKcu1500
make
...
Flashing KCU1500 with Firmware
configuration memory part number for kcu1500
mt25qu512-spi-x1_x2_x4_x8
https://githubdocs.google.com/presentation/slaclabd/cameralink-gateway1VVfkIWN9M_czZiaXhK4iFp-Drj_yc64smbzpSwZ61Cg/edit?usp=sharing
Front-end Board (pgp->camlink converter) Firmware
https://github.com/slaclab/cameralink-gateway
For the OFFICIAL BOARD use configuration memory part number s25fl128sxxxxxx0-spi-x1_x2-x4
For the EVAL BOARD When programming the front-end board firmware (a xilinx KC705 rev. 1.1) use configuration memory part number 28f00ap30t-bpi-x16
Making Vivado communicate with board over USB/JTAG
Here's the mcs file location
$TOP/cameralink-gateway/firmware/targets/ClinkFebPgp2b_1ch/images/ClinkFebPgp2b_1ch-0x00000025-20190315182526-ruckman-af9cde50.mcs
Making Vivado communicate with board over USB/JTAG
Larry has some Larry has some slides on how to program the flash chips (mt25qu512) on the KCU1500 via USB/JTAG. Startup "vivado" after setting up the firmware
...
SLAC Driver
Build/install the datadev.ko driver using the instructions here:
...
This needs to be done on the machine where the KCU1500 lives. You need sudo on the machine to install the driver.
Programming FPGA over PCI
After the first programming (and power-cycling) described above, use this script to reprogram:
...
(computer lcls-pc83236 loads the driver on boot using /etc/sysconfig/modules/datadev_0.modules as of 7/10/2018)
TimeTool Software Files
lcls2-pcie-apps/firmware/applications/TimeTool/python/TimeTool.py: a description of the "addValue" register
...
- self.add adds registers and associated GUI control
Settings Needed To Run Camera
Use channela to talk to the front-end board in the "Variables" tab:
...
Code Block |
---|
Got Response: Got Response: Model P4_CM_02K10D_00_R Got Response: Microcode 03-081-20296-13 Got Response: CCI 03-110-20294-03 Got Response: FPGA 03-056-20470-03 Got Response: Serial # 12102856 Got Response: BiST: Good Got Response: Got Response: DefaultSet 1 Got Response: Ext Trig Off Got Response: Trig Overlap Off Got Response: Line Rate 1 [Hz] Got Response: Meas L.R. 6 [Hz] Got Response: Max L.R. 19607 [Hz] Got Response: Exp. Mode Timed Got Response: Multi Exp. Mode Off Got Response: Exp. Time[0] 50000 [ns] Got Response: Meas E.T.[0] 50000 [ns] Got Response: Max E.T. 3000500 [ns] Got Response: Got Response: Test Pat. 1:Ramp1 Got Response: Direction Internal, Forward Got Response: TDI Stages 2 Got Response: Vert. Bin 1 Got Response: Hor. Bin 1 Got Response: Flat Field Off Got Response: Offset 0 Got Response: System Gain 1.00 Got Response: Mirror Off Got Response: AOI Mode: Off Got Response: Scan Type Line Scan Got Response: CL Speed 85MHz Got Response: CL Config Medium Got Response: Pixel Fmt 8 bits Got Response: CPA ROI 1-2048 |
Notes From Matt On Evr Firmware
wrapper to transceivers:
https://github.com/slaclab/lcls-timing-core/blob/release-lcls2/LCLS-II/core/rtl/TimingGthWrapper.vhd
...
axi-stream: amba-xilinx-interconnect: no address involved, like a port, push/acknowledge
axi: full memory interface: address/values and can burst multiple values
axi-lite: used for register interfaces: 32-bit value with address
What we learned about timing stream:
eventcode is part of TimingStreamType
timingstreamrx (timing message) and timingrx (timing message) have outputs of type timingstream
timingrx instantiates both LCLS-I and LCLS-II timing streams:
...
Everything in the following lines are axi extensions called (slac streaming interface) ssi
tDest : slv(7 downto 0); identifies the destination for when using multiple axis on a single bus
tId : slv(7 downto 0); transaction id for handshaking (validation the signal was )
tUser : slv(127 downto 0); user bits. can be used for anything. have been used for start of frame (SOF) and end of frame errors (EOFE),
end record AxiStreamMasterType;
Thoughts on Controlling Hardware Trigger From Ryan (04/11/18)
- Should not mock-up an axi-write every time we receive the right event-code
- Currently the SWControlEn/SWControlValue are direct axi writes to front-end board
- Will need to change front-end board firmware
- TxControl in KCU1500 has 8-bit field and valid that needs to be set to communicate to front end board. Will map this to CC in front-end board.
- TxControl goes through Pgp2bAxi module.
- Can control these registers both with software (for debugging) or hardware.
Tracking Down Timing Fiber Input
- pgplanewrapper.vhd has this code:
evrRxP(1) <= qsfp1RxP(3);
evrRxN(1) <= qsfp1RxN(3);evrRxP(0) <= qsfp1RxP(2);
evrRxN(0) <= qsfp1RxN(2);
- not sure why 2 EVRs (maybe lcls1 and lcls2?)
- other 6 lanes are all hooked up to PGP
- pgplanewrapper also hooks up the 6 dmaObMasters/dmaIbMasters etc to the pgp lanes (last two are unused, because they are used by evr)
- timetoolkcu1500.vhd hooks dmaIbMasters(0) to the AxiStreamTap, so our camera data is on qsfp0[0], which is the first of the 6 pgp lanes
- timing fibers are on the last two fibers of qsfp1
Setting Up TimingCore
- under "commands" tab need to ConfigLclsTimingV1. this controls a multiplexer that routes the evr clock/data to the fpga, either from lcls1 or lcls2
- under "variables" tab GtLoopback 0 is normal mode, 2 is internal loopback which includes EVG simulator sending some opcodes, 4 is a later loopback as described on page 85 of Xilinx ug576 guide https://www.xilinx.com/support/documentation/user_guides/ug576-ultrascale-gth-transceivers.pdf
- mmcm is like a souped-up old dcm (digital clock manager) able to generate many more frequencies using, with jitters ~25ps (comparable to a standard external oscillator)
- each of qsfp1 input 2 (counting from 0) can only accept lcls1 timing (static) and input 3 can only accept lcls2 timing. Possibly it's switched, but I don't think so.
- in "variables" tab TimeToolDev->HW->TimingCore->EvrCore, sofCount/eofCount/FidCount should all increment at 360Hz if things are working.
Using Xilinx Integrated Logic Analyzer (ILA, aka "ChipScope")
- use the vivado core-generator to generate an ila_0 core. Matt says use the generated .xci file, not the .dcp file
- add the .xci file to appropriate ruckus.tcl with something like this:
- loadIpCore -path "$::DIR_PATH/coregen/ila_0.xci"
- instantiate an ila_0 component in vhdl
- build and program the fpga
- open vivado (currently on rdsrv223)
- connect over jtag
- find .ltx file generated by build with ila. this is a list of signals that are exported to the ila.
- in vivado click on the fpga (e.g. xck115_0) in top left
- in the "hardware device properties" enter the .ltx file in the "probes file" field
- a list of ila's should appear along with waveforms in the display
...
- also needed:
- under "commands" tab need to ConfigLclsTimingV1. this controls a multiplexer that routes the evr clock/data to the fpga, either from lcls1 or lcls2
see ILA core generation visual step through for walk through on how to setup.
Simulation tools
...
how to use pyrogue from python shell
To do list
...
gui fully removed,- still need to make smart and stream line
...
- spy on the timestamp multicasts in software (maybe not necessary)
take real pictures with lens from Ryantrigger delay and event-code programmable via AXIeliminate gui and save hdf5put all setup commands in python scriptdo we need feedback for laser? (no)- run at 120Hz overnight and validate
debugging timing in firmware
look for .rpt files in the
build/TimeToolKCU1500_project.runs/impl_1/*.rpt
and in
build/TimeToolKCU1500_project.runs/synth_1/
grep -A 3 VIOLATED TimeToolKcu1500_timing_summary_routed.rpt | grep "Destination:"
presentation / demo
visual notes
Matt Weaver's LCLS2 time stamping
https://github.com/slaclab/l2si/blob/master/firmware/common/base/rtl/EventHeaderCache.vhd
AxiStream Batcher Protocol Version 1
To do list
prescaling data by x amount over registersimulation of modules inside TimeToolCore.vhd- peak-finding (final parameters)
- programming weights by axi-stream
- emulate LCLS2 timing system in firmware
- divide by the delayed-minus-pedestal (use LUT?)
virtual evg. (should only require gui tweaks. check with ILA)- how to handle the git large files
get rid of guigui fully removed,- still need to make smart and stream line
programmable event-code & trigger delay (axi bus)emulate epix model (send timestamp to front-end board. time stamp should be at the beginning of image packet.). later: maybe this isn't necessary because in future we would support camlink-over-fiber cameras with no front-end board?save hdf5full/deca mode (only 8 bit for deca or full)get test stand in 901 working- start with Matt matched filter algorithm or Abdullah algorithm
- feedback results to Joe Frisch as udp packets or accelerator-style-pgp
is 8-bits OK for deca? (answer: yes, for interferometry mode)- understand how system behaves at high rates (do we drop frames? timestamps correct?)
- reuse matt's firmware for timestamping
- add in LCLS2 timing (triggering)
- send full signal to Matt
- switch from python to C
- think about running the camera all the time (to send feedback data continuously)
- difference kc705 revisions 1.1 vs 1.2.: no significant difference according to https://www.xilinx.com/support/answers/59751.html. Either an FMC problem, or subtle timing issue, or need to specify board rev somehow when synthesizing?
- interface to DRP?
new front-end boardstouch base with Ryan on code structure issues- prepare for running in LCLS-I? If yes:
- spy on the timestamp multicasts in software (maybe not necessary)
take real pictures with lens from Ryantrigger delay and event-code programmable via AXIeliminate gui and save hdf5put all setup commands in python scriptdo we need feedback for laser? (no)- run at 120Hz overnight and validate
resolve vhdl axi-lite offset constantswhy doesn't offset 0x400000 (which turns into 0xc00000) work?https://forums.xilinx.com/t5/Synthesis/How-to-display-post-synthesis-integer-constant-values/td-p/692285
Algorithm Conversation With Giacomo (March 6, 2019)
- Giacomo is doing spatial, Ryan is spectral (roughly speaking)
- both are interferometric, so the background we had in LCLS1 is mostly gone, but still have some background from stray light, for example
- Formula that should work for both: (S-B)/(S(delayed)-B(delayed)). (note: division is similar to normalized subtraction: (S-B)/B = (S/B)-1
- B is different for S and S(delayed) (background at time "delayed" is taken)
- Giacomo thinks we won't need the "delayed" subtraction/division for 80% of the experiments
- Giacomo will ensure that we run in a regime where (S(delayed)-B(delayed)) is not close to zero
- sxrx34917 (being analyzed by Giacomo and Stefan Droste). this data is "zoomed out" in the time dimension, but will be more zoomed-in for LCLS-II.
- S(delayed) comes from moving the laser out of the time window, and is done "offline" according to Giacomo every few hours
- Giacomo wants a fit to the minimum of dI/dt (derivative of I-B) to a parabola
- Giacomo thinks that the feedback will be slower (millisecond level). Contradicts what we learned before where Joe Frisch was going to do fpga stuff with the laser-locking system
- algorithm should be same for spatial/spectral
- Giacomo wants a way to look at S-Delay when the timing edge is out of the window. Tells them if it's on the "right" or the "left".
- Questions for Giacomo:
- the edge looks big in sxrx34917, do we definitely need to divide? Answer: no, as long as we can set an ROI. i.e. don't need
- does the "delayed" contain background from stray light, for example? Answer: "delayed" is the pump laser delayed off the end of the camera image
- is B just pedestal, or does B include laser-pump? If it's laser-pump, do we need to use IIR to compute B? Answer: B is remaining background (same as LCLS-I) not eliminated by interferometric approach.
- do we use the same B in the numerator/denominator? Answer: no
Conversation with Ben Reese (2/13/20)
feb fpga version on feb. 13 2020:
ClinkFebPgp2b_1ch built Tue 16 Jul 2019 10:39:15 AM PDT by ruckman on rdsrv222
_TimeToolKcu1500Root.py has register settings
https://github.com/slaclab/lcls2-timetool/blob/master/software/config/config_20200207_152212.yml
ConfigureXpmMiniSim
TimingRx->TimingPhyMonitor->UseMiniTpg (selects minitpg)
(check clock freq's under TimingPhyMonitor)
switching to real timing need an xpm txreset: see this as RxClkFreq not being the same at TxClkFreq
pipelinedepthfids=90
config_l0_select_ratesel=0x7
7 = 1Hz
5 or 4 is 100Hz
confg_l0_select_enabled=1 starts triggers
TriggerEventManager->TriggerEventBuffer[0]
- has l0 counts
- has xpmpause/xpmoverflow (too much data). xpmoverflow is a sign backpressure not working well enough: reduce pausethreshold. xpmoverflow is latched version of fifooverflow. permanently latch deadtime. needs a clearreadout to reset.
- missing clearreadout going to front end board
- pausethreshold sets when to send deadtime to XPM
XpmMessageCount increments @1MHz
clinkfeb[0]->->clinktop->trigcontrol[0] has a trigger rate
TriggerEventManager->TriggerEventBuffer[0]->masterenable enables triggers going to frontend
TriggerEventManager->TriggerEventBuffer[0]->eventbufferenable enables the fifo that receives the timing frames (and sends the\
m to the batcher eb)
mostly masterenable/eventbufferenable should go together?
eventbufferenable first, masterenable second
xpmmini is only partition(rog) 0.
to switch to realxpm:
- Kcu1500Hsio->TimingRx->ConfigLclsTimingV2
- toggle UseMiniTpg to false
watch kcu1500hsio->timingrx->timingframerx->RxDown (resets to 0 on ConfigLclsTimingV2, latches when link drops)
todo:
- clear readout (workaround TimeToolKcu1500->Application->AppLane[0]->EventBuilder->Blowoff)
- trigger 10us feedback
- edge calculation
...