Group C SVT Setup

Useful Links

SVT EPICS Power Instructions

DAQ & Power Mapping in Group C

Operation

!!! Remember to make sure the detector is being cooled. !!!

!!! Remember to make sure the interlocks are in the state you want them. !!!

Logbook

1/14/2013

System status (from Ben's email Jan 7th):
We can currently power 8 FEBS.  We can power every hybrid attached to those 8 FEBS: 30 Hybrids.  All Hybrids that we can power are able to sync, except for FEB 8, Hybrid 2, which has just 1 APV that won’t sync.
Link to excel file. 

L1t (DAQ FEB #0 and GUI FEB #0) has what looks like a short on the output side of the regulator on DVDD (we can’t see the short probing the pins from outside).
L6b (DAQ FEB #5 and GUI FEB #7) first appeared to have a short on DVDD. After 5mins and a power cycle it came back with current levels that looked reasonable except that the DVDD level indicated it had no control link. 

10:00AM
Danh Du finished terminating the pigtails from the flange boards. We are missing a single female pin on one REDEL connector. Its wire was cut and folded back waiting for new pins. He did not test the pinout at this point.

11:00AM
Removing FEB cooling plate from SVT box to rework FEBs.

12:54PM
Extracted thee FEB cold plate and removed all FEBs. The two non-working FEBs (L1t and L6b) were sent to Ben directly; the other were given to Lupe for rework.
The default rework is to add Zener diodes to all FEBs and also to remove the FPGA LED that is always on (it's on the cooling plate side so we can't see it anyway; it indicates FPGA programming OK).
I added the serial number of each board to the mapping on the DAQ & Power Mapping in Group C page.

4.00pm
Found that P5 connector had DIG_GND swapped across the flange. The two pigtail connector seemed ok (the twisted pair went into adjacent pins as expected) which leaves the only possibility of having a swap at the pads on either side of the flange board.
=>Danh Du corrected this on the air side P5 connector.

5.15pm
Placed FEB L1t on cold plate next to the SVT box and connected P1 power cable. Without detector attached the FEB powers up fine with expected currents. When connecting the L1t detectors we see again a 3A current on DIG as before. We also tried connecting L1-3 spare half-modules with the same result. We then swapped out P1 with P5 and powered the FEB as if it was L1b. This was successful. Thus the problem points to some problem in the L1t power cable. Tomorrow we need to ohm out that path to see if we can find a problem.

6:30 p.m.
L1t, L6t and L6b modules have been tested with the single-FEB setup. All OK.

1/15/2015

11:00am
Ohming out the L1t power connection revealed a short on the long cable between the breakout boxes between DIG power and the metallic braid. This together with the fact that the SVT box, and therefore hybrid grounds, was not isolated from the optical table most likely caused a current path from the braid to the breakout box and onto the detector. This is why connecting the FEB to hybrids made the difference; it is otherwise isolated from the SVT box. 
The long cable (P1) is being reworked now and ready today most likely.

12:00pm
The L6b FEB was mounted and the link was working fine. To fully test it we slide it in to be able to connect to the right detectors and it is still fine. Data cable not seated correctly might be the explanation. We didn't power any other FEB that share the same flange to see if that was an issue but given that other FEBs worked means we will take that risk. 
The plan is to extract these two boards and get them reworked. Then have all 10 FEBs and the octopus cleaned and mounted back on the cooling plate as soon as possible. Before sliding the cold plate in we would like to power all FEBs and walk through each one with a set of spare hybrids to make sure things are working.

1/16/2015

9:45am
Finished Ohming out the HV connections from the PS to the last REDEL connecting to the patch panel. No issue was found. What is left to do is to verify the GUI to physical hybrid mapping. This will be done once we have the patch panel attached.

2:45pm
Connected HV to the patch panel and mapped out the GUI to physical hybrid channel. This mapping was added to the DAQ & Power Mapping in Group C page. I will update the HV GUI with a physical hybrid ID (layer, hybrid channel instead of the existing mapping).

1/17/2015

1:00 pm - 5:00 pm

With all FEB's mounted on the support plate, the FEB's were powered up individually with the high speed cables connected.  All FEB's seem to power up fine, however, the DVDD current on FEB 5t and FEB 6t seem to indicate that there was no control link present.  These two FEB's were connected to the top two channels (0,1) of the 4th flange board using cable #10 and #11 respectively.  In order to check whether the problem was due to the FEB's or the cable/flange, a working flange board channel (flange board 3, channel 2, cable #6) was connected to FEB 5t and it was powered up again.  This time, a control link was established so the problem seems to be with either the cable or the flange board.  The same procedure was used to test FEB 6t and the results were the same, however, a different flange board and channel were used (see below).

Once a control link was established with all FEB's, they were tested to determine if they could sync all APV's connected to them.  This was done by using a stack of 4 L1-L3 hybrids connected to a single crossover board which in turn was connected to a single FEB.  The same stack of hybrids was cycled through all FEB's and only the FEB being tested was powered on.  As shown on the table below, only 5/10 FEB's were found to sync all 20 APV's.

The clock phases were not adjusted on those FEB's which couldn't sync all 20 APV's, so it's unclear if this would resolve the issue.

1/19/2015 

10:00AM

Connected all FEBs to power and data as on Saturday. Verified by powering up all boards are consistent with Saturday. Ben will work on trying to sync all APVs during the day.

11:00AM

When mounting the boards on Friday 1/15 the top and bottom FEBs were swapped for L1 and L2-3. Since there is no particular reason yet for having a specific FEB processing a particular layer we will keep this as the new layout.  Therefore all testing on Saturday was with this new layout. This has been updated on the DAQ & Power Mapping in Group C page.

11:00AM-17:00AM

Status update for today:
 We are able to get sync on all of the FEBs with the stack of spare hybrids. We might have had an issue with not setting the power configuration correctly yesterday which means that the hybrids might have gotten up in some funny state. It’s unclear, but doing it the right way seem to solve all the issues. 

All flange channel were working. The issue with the two channels we didn’t see any control link on was a firmware issue. On the control dpm we only accepted 10 links and the channels that we were using happened to be on two unused channels. Ben changed the firmware to accept 12 inputs and things got solved. We put the flange back together again after that as it was. 

 We are ready to put the FEB cooling plate into the detector and hook up the detectors to continue testing. 

1/20/2015

FEB plate was installed and hooked up today.

We see one problem with FEB power - L5b DVDD sense is not connected (sense voltage 0, terminal voltage maxed out), and this comes and goes as the pigtail is manipulated. Opening up the backshell on the flange pigtail, the DVDD+ sense pin (pin 20) is clearly not terminated correctly - you can see the stripped end of the wire pulling out of the connector, it looks like a poorly crimped pin or something. FEB power looks fine otherwise. Good news.

We don't get any data link on two FEBs - L4t and L4b. The DVDD current on these FEBs doesn't rise when the flange is turned on. Since these are the two hardest FEBs to plug in, we suspect the high-speed cables are not plugged in right on those FEBs. Unless we find some other cause for this problem, it means opening up the box again. But if we're right, it's an easy fix. We do get link on all the others, so Ben's firmware fix is working.

We can't turn the hybrids on - Ryan's Python script gives a socket error. This is probably a software issue. Nothing to worry about, but it means we didn't get to check sync or take data tonight.

We left the cooling running, and the MPOD off.

1/21/2015

The poorly crimped pin (pin 20) on pigtail P4 corresponding to DVDD+ sense was repaired by Danh Du.  Shortly after, FEB L5b was powered on and the previously observed power issues (sense voltage = 0 and terminal voltage maxed out) were no longer present.

The SVT box was opened up and the high speed cables connected to L4t and L4b were re-seated.  Once this was done, a control link was established with FEB L4t and L4b. 

It was noticed that the current drawn by DVDD on FEB L6b (FEB 15) was lower than expected.  After probing the board, it was determined that the A+2.9V_DVDD regulator circuit had blown up.  The FEB support plate was then removed from the SVT box and the troublesome FEB was replaced by FEB 13.  Once the FEB support plate was reinserted into the SVT box, FEB L6b was powered on and everything looked as expected.

After reconnecting all of the high speed cables and power, all mounted FEB's were powered on and the control links on all of the were checked.  This check took place after the new firmware which allowed the control DPM to receive all 12 links had been loaded.  The test showed that all FEB's had control links, however, the control link associated with FEB L6b had several errors.  This is a previously known problem that can be fixed by switching FEB L6b to another flange channel.  The data links also looked fine. 

Once all FEB's were verified to work, the SVT box was sealed up, the FEB support plate cooling was connected and the chillers were turned on.  The cooling was left on and power was left off for the night.

1/22/2015

5:00PM System status

Detector fully hooked up.
FEB chiller at +20C.
SVT chiller at +20C.   
FEB power looks good on all boards.

Summary:

Two FEBs have issues. Including notes on those below.

Physical FEB: L4b Serial: 8 FebFpga: 8 Fiber: F
==============================================
DataDPM 0, DP 0, Hybrid 3, all sync 
DataDPM 0, DP 1, Hybrid 2, all sync 
DataDPM 0, DP 2, Hybrid 1, all sync
DataDPM 1, DP 0, Hybrid 0, 4/5 synced

All of these synced initially, but after trying to resync, they began to have issues. The APV's on Hybrid 0 seem to drop in and out of sync. After power cycling, the issues persist with Hybrid 0. The peak and base values on APV 4 look off. After playing with the AdcClkPhaseShift and Data delays and setting them back to what they initially were, the sync and base values on APV 4 improved. We will know if there is an issue with the data after a run is taken.

Physical FEB: L6b Serial: 13 FebFpga: 6 Fiber: F 
==============================================
Control link has several errors which don't allow communication to the board. Switched control fibers E and C at the COB. The FPGA number switched to 3. There were no data links present. Looking at the oscillator that is used to driver the links, the frequency is at half of what is it suppose to be.

Detailed log on each FEB below:  

Physical FEB: L1t Serial: 9  FebFpga: 2 Fiber: A
==============================================
DataDPM 0, DP 0 (No hybrid connected)
DataDPM 0, DP 1 (No hybrid connected)
DataDPM 0, DP 2, Hybrid 1, all sync
DataDPM 1, DP 0, Hybrid 0, all sync

Physical FEB: L2-3t Serial: 14 FebFpga: 5 Fiber: D
==============================================
DataDPM 4, DP 0, Hybrid 3, all sync
DataDPM 4, DP 1, Hybrid 2, all sync
DataDPM 4, DP 2, Hybrid 1, all sync
DataDPM 5, DP 0, Hybrid 0, all sync

Physical FEB: L4t Serial: 5 FebFpga: 9 Fiber: H
==============================================
DataDPM 1, DP 3, Hybrid 2, all sync
DataDPM 3, DP 3, Hybrid 1, all sync
DataDPM 5, DP 3, Hybrid 0, all sync
DataDPM 6, DP 2, Hybrid 3, all sync

Physical FEB: L5t Serial: 11 FebFpga: 10 Fiber: H
==============================================
DataDPM 5, DP 1, Hybrid 3, all sync
DataDPM 5, DP 2, Hybrid 2, all sync
DataDPM 6, DP 0, Hybrid 1, all sync
DataDPM 6, DP 1, Hybrid 0, all sync

Physical FEB: L6t Serial: 10 FebFpga: 11 Fiber: H
==============================================
DataDPM 4, DP 0, Hybrid 3, all sync
DataDPM 4, DP 1, Hybrid 2, all sync
DataDPM 4, DP 2, Hybrid 1, all sync
DataDPM 5, DP 0, Hybrid 0, all sync

Physical FEB: L1b Serial: 7 FebFpga: 0 Fiber: A
==============================================
DataDPM 2, DP 2 (No hybrid connected)
DataDPM 3, DP 0 (No hybrid connected)
DataDPM 3, DP 1, Hybrid 1, all sync
DataDPM 3, DP 2, Hybrid 0, all sync

Physical FEB: L2-3b Serial: 6 FebFpga: 3 Fiber: D
==============================================
DataDPM 1, DP 3, Hybrid 2, all sync
DataDPM 3, DP 3, Hybrid 1, all sync
DataDPM 5, DP 3, Hybrid 0, all sync
DataDPM 6, DP 2, Hybrid 3, all sync

Physical FEB: L4b Serial: 8 FebFpga: 8 Fiber: F
==============================================
DataDPM 0, DP 0, Hybrid 3, all sync
DataDPM 0, DP 1, Hybrid 2, all sync
DataDPM 0, DP 2, Hybrid 1, all sync
DataDPM 1, DP 0, Hybrid 0, 4/5 synced

All of these synced initially, but after trying to resync, they began to have
issues. The APV's on Hybrid 0 seem to drop in and out of sync. 

After power cycling, the issues persist with Hybrid 0.

The peak and base values on APV 4 look off. After playing with the
AdcClkPhaseShift and Data delays and setting them back to what
they initially were, the sync and base values on APV 4 improved. We will
know if there is an issue with the data after a run is taken.

Physical FEB: L5b Serial: 12 FebFpga: 7 Fiber: F
==============================================
DataDPM 1, DP 1, Hybrid 3, all sync
DataDPM 1, DP 2, Hybrid 2, all sync
DataDPM 2, DP 0, Hybrid 1, all sync
DataDPM 2, DP 1, Hybrid 0, all sync

Physical FEB: L6b Serial: 13 FebFpga: 6 Fiber: F
==============================================
Control link has several errors which don't allow communication to the board.

Switched control fibers E and C at the COB. The FPGA number
switched to 3.

There were no data links present. Looking at the
oscillator that is used to driver the links, the
frequency is at half of what is it suppose to be.

1/23/15 - TKN

Checked the grounding scheme of the detector.

L1-3 u-channel / SVT Box: 0.1 Ohms

FEB cooling plate / SVT Box: 6.7 Ohms

FEB cooling plate / table: 4.6 Ohms

SVT Box / table: 3.4 Ohms

Seems to imply that the table is between the FEB cooling plate and the SVT Box.  Perhaps implies a connection from FEB cooling plate to vacuum box (e.g. via data cable edge contact) and then vacuum box to table then table to SVT Box?

We should try to break this when we can.

6:40pm

The hybrid power monitoring and control now works through the hybrid GUI. It's mapped using feb index. Getting the physical name for it on the GUI is a little tricky but I'm working on it so until then you need to find out which feb nr you have powered (either from the non-zero hybrid currents that are updating, the temperature GUI or the daq GUIs). 

1/30/15

10:00AM - PH
Testing L4b (serial #15) mounted on production plate placed on the test bench. Using one of the spare flanges that are not potted.The FEB ID is 2, data "DPM:channel" is 0:0, 0:1, 0:2 and 1:0. I see a few data link errors on 0:2. All hybrids sync. Took ~5000 events, file name is test.dat_000785.evio.0.

10:30AM - PH
Testing L1t (serial #01) with same setup as the L4b above. All hybrids sync. Took ~6400 events, file name is test.dat_000787.evio.0.

11:50AM - PH

There is a broken pin on L1t (serial #01). That particular one is not used. Retesting with same setup as the L4b above.  All hybrids sync. Took ~4800 events, file name is test.dat_000789.evio.0.

Equipment

Interlock board

see Group C interlock board for instructions on monitoring and controls.

Serial portserver

Digi Portserver TS 4: 192.168.1.217 on LAN. Telnet from ppa-pc91245, e.g. "telnet 192.168.1.217 2001" will connect you to port 2001. All serial ports are logged to /var/log/portserver on ppa-pc91245. Configure the portserver using its web interface or Telnet. Docs:  http://www.digi.com/support/productdetail?pid=1954

Thermoelectric chiller (backup chiller for FEB)

Oasis 190: manual

Error bits:

Procedures

Basic instructions to take a pedestal run with CODA

To be added.

Miscellaneous

Reloading PGP card drivers

Typically after a reboot we need to reload the pgp card driver. 

$ cd /u1/pgpcard/software/driver_old/

$ ./ pgpcard_load

OLD:Take a calibration run with expert SVT GUI

Start the DAQ:

$ cd /u1/software/software_new

$ source setup_env.csh

$ ./bin/frontEndTestGui

Click <read status> and make sure no error is seen. If so, check that the FEB is powered correctly.

Make sure the AxiXadc temperature is ok (50-60C).

Configure with config/FrontEndBoardConfigC01.xml

Click <read status> and check that more registers are found.

Turn on power to hybrids (see below)

Do a hard reset under commands/FebCore/HybridHardReset (current on AVDDP should drop ~100mA).

Click <Write config>.

Do a soft reset under commands/FebCore/HybridSoftReset.

Check that the hybrid currents measured are ok. Sync status for each channel can be seen in register RceCore/DataPath[x]/Synced, 1xf for powered channels.

Test Run SVT Group C Setup

Group C SVT test setup