This page describes the LSST version of the PGP Card Gen2 driver. This version of the driver was setup to support additional features required for the LSST project. 

These new features include:

  • Ability for separate processes / threads to access each lane/VC using a mask.
  • Ability to transmit and receive data blocks larger than compiled DMA buffer size using continue flag.
  • Ability to send and receive 8-bit side band information

This version of the driver will be converted to the baseline driver and may be renamed to PgpDriverMask.

Getting The Driver

The driver can be obtained in the SLAC subversion repository:

> svn checkout file:///afs/slac/g/reseng/svn/repos/pgpcard/trunk/software/lsst_driver

Or from your contact at SLAC.

Building The Driver

Go to the driver sub-directory and build:

> cd driver

> make

You will need to make the appropriate kernel headers and development packages installed. Contact rherbst@slac.stanford.edu if you have any issues. The kernel has been compiled in all recent versions of the Linux kernel.

Installing The Driver

A script exists in the driver sub-directory to install the driver and create the appropriate device files (future versions of the driver will self-create the device files, but this has not yet been added). 

> ./pgpcard_load

This will create the file /dev/pgpcard0 for driver access.

Opening The Card

Accessing the PGP Card driver makes use of a set of inline wrapper functions defined in include/PgpCardWrap.h. This file must be included in applications along with include/PgpCardMod.h which includes the structures shared between the driver and user space code.

Example code for opening the device:

s = open("/dev/pgpcard0", O_RDWR);

The call will return a file descriptor if the operations success or -1 on failure. Multiple devices are allowed to open a pgpcard device.

After opening you will need to set the mask for the channels you will be receiving data for:

> pgpcard_setMask(s,0xFFFF);

The above call will allow reception for all 4 virtual channels on all 4 physical lanes supported by the PGP Card. The mapping of this mask is:

  • Bit 0 = Lane 0, VC 0
  • Bit 1 = Lane 0, VC 1
  • Bit 2 = Lane 0, VC 2
  • Bit 3 = Lane 0, VC 3,
  • BIt 4 = Lane 1, VC 0,
  • Bit 15 = Lane 3, VC 3

Currently the driver only uses the mask when receiving data. All processes/threads can write on all lanes and virtual channels.

The card then be closed using a standard close command

> close (s)

Any reserved lane/VCs will be released and the associated queues will be drained and released.

Sending And Receiving Data

Once the port has been opened and the proper mask has been set the application can send and receive frames using the PGP Card.

Sending

The declaration for the send function is:

inline int pgpcard_send(int fd, void *buf, size_t size, uint lane, uint vc, uint cont=0);

The passed args are:

  • int fd = file descriptor from previous open
  • void *buf = Buffer containing data to send
  • size_t size = Number of 32-bit values contained in buffer. 
  • uint lane = Lane to send data on
  • uint vc = Virtual channel within the lane to send data on
  • uint cont = Set to '1' if this is not the last transfer in a frame. Normally the application can set this to '0'.

The return value is the total number of 32-bit values transferred or a negative number on error.

  • -1 is returned if there is an error
  • -EAGAIN is returned if the buffer was not ready

Receiving

The declaration for the receive function is:

inline int pgpcard_recv(int fd, void *buf, size_t maxSize, uint *lane, uint *vc, uint *eofe, uint *fifoErr, uint *lengthErr, uint *cont=NULL);

The passed args are:

  • int fd = file descriptor from previous open
  • void *buf = Buffer for received data
  • size_t maxSize = Maximum number of 32-bit values that can be received.
  • uint *lane = Pointer to integer which will be updated with the lane associated with received data
  • uint *vc = Pointer to integer which will be updated with the virtual channel associated with received data
  • uint *eofe = Pointer to integer which will be set to '1' if there was an EOFE received with the frame
  • uint *fifoErr = Pointer to integer which will be set to '1' if there was a FIFO overflow errror (flow control) in the frame
  • uint lengthErr = Pointer to integer which will be set to '1' if there was length error. Either the frame was larger than the allowed buffer or the user buffer was too small.
  • uint cont = Pointer to continue flag. Will be set to '1' if more frame data is expected.

The return value is the total number of 32-bit values received or a negative number on error.

  • -1 is returned if there is an error
  • -EAGAIN is returned if the buffer was not ready

Additional Functions

The PgpCardWrap.h file contains additional functions which can be used:

inline int pgpcard_status(int fd, PgpCardStatus *status);

Updates the passed status pointer with the current status of the card. The structure of the status record can be found in PgpCardMod.h

inline int pgpcard_setDebug(int fd, uint level);

This function enables addition debugging to the kernel log. Normally not used by users.

inline int pgpcard_setRxReset(int fd, uint lane);

inline int pgpcard_clrRxReset(int fd, uint lane);

inline int pgpcard_setTxReset(int fd, uint lane);

inline int pgpcard_clrTxReset(int fd, uint lane);

Assert or clear the receive or transmit reset for a PGP lane. Used when the receiver or transmitter won't link or is not behaving. Usually not needed in normal operation.

inline int pgpcard_setLoop(int fd, uint lane);

inline int pgpcard_clrLoop(int fd, uint lane);

Enable or disable loopback for a particular lane. Setting this will allow a PGP port to link to itself. Any sent frames will be received on the same lane.

inline int pgpcard_setData(int fd, uint lane, uint data);

Set the currently transmitted side band data for a lane. The currently received sideband data can be found in the status record. This is not used in most applications.

inline int pgpcard_rstCount(int fd);

Reset all counters reported in the status record.

inline int pgpcard_dumpDebug(int fd);

Dumps internal buffer states to the kernel log.

Status Record

The following fields are available in the status record:

  • __u32 Version = Current firmware version of PGPCard
  • __u32 Pgp#LoopBack = Current loop back status for a lane
  • __u32 Pgp#RxReset; = Current RX reset state for a lane
  • __u32 Pgp#TxReset; = Current TX reset state for a lane
  • __u32 Pgp#LocLinkReady; = Local PGP core is showing link
  • __u32 Pgp#RemLinkReady; = Remote PGP core is showing link (only valid if local is linked)
  • __u32 Pgp#RxReady; = Local RX PHY is ready
  • __u32 Pgp#TxReady; = Local TX PHY is ready
  • __u32 Pgp#RxCount; = Number of received frames for a lane
  • __u32 Pgp#CellErrCnt; = Number of cell errors for a lane
  • __u32 Pgp#LinkDownCnt; = Number of link down events for a lane
  • __u32 Pgp#LinkErrCnt; = Number of link errors for a lane
  • __u32 Pgp#FifoErr; = Number of FIFO overruns for a lane
  • __u32 Pgp#RemData; = Current received side band data for a lane
  • __u32 Pgp#RemBuffStatus; = Flow control status of remote end of the link

All other fields are reserved for development use.

Distributed Applications

There is a sub directory called app in the distributed driver software. This directory contains a number of applications that are used to test and monitor the PGP Card. These applications can be compiled by typing 'make' in the app directory.

  • xstatus /dev/pgpcard0 = Will read and display the current status record from the PGP Card
  • xloop lane value = Will set or clear the loopback of a PGP lane. Assume /dev/pgpcard0 is the device. 
    • xloop 0 1 = Sets loopback for lane 0
    • xloop 0 0 = Clears loopback for lane 0
  • xrate2 = Will test the card by generating write and read routines for all 4 ports. An optional arg defines the number of virtual channels to test. All four ports must be set in loopback mode for this application to run.
  • xPromLoad = Will program the PgpCard firmware. Use only under the direction of the developer.
  • xPromVerify = Will veriify the PgpCard firmware. Use only under the direction of the developer.

 

 

 

  • No labels