Overview

Two commercial vendors competing for LSST business have been awarded contracts for a set of preliminary devices.  Before shipment, each device will undergo a series of tests by the vendor, producing ~25 GB of data.  These data must be transferred to SLAC, analyzed, archived and distributed to other LSST laboratories prior to a "pre-ship review".  Only after successfully passing this review will a device be authorized for shipment.  It is expected that SLAC will receive multiple data deliveries per month although they are not precisely scheduled and deliveries may be bundled.

Requirements

• Vendors must be able to control the timing of the data delivery.  
  • The intranet within the company as well as their connection with the internet can be overwhelmed by these deliveries
  • Vendors insist on controlling the exact timing for the electronic data transfers
• Vendors must be able to restartan aborted data delivery.
  • The internet connection between SLAC and the two vendors is subject to intermittent instability.
  • The internet connection to at least one of the vendors is quite slow (100 Mbps)
  • The need to restart a large and time-consuming delivery from scratch would cause an unacceptable delay
• Vendors must be able to create, modify, or delete files in their FTP areas
• The transfer buffer must be able to hold multiple data deliveries per vendor, so at least 200 GB
• LSST must do its best to prevent data from Vendor A from being visible to Vendor B, and vice versa

Proposed Solution

• LSST operated advanced FTP service
  • vsftp server software: very secure, high performance, restartable transfers, private ftp-only accounts
  • installed and running on LSST service VM (VM is "SCS Standard")
• New FTP server is configured to have ownership privs on a single NFS partition: /nfs/farm/g/lsst/u2 (which will be a short-term buffer from which a permanent archive will be made)
• Individual vsftp accounts for Vendors A and B.  

Potential Security Issues and Mitigations (not complete!!)

1. Hacking into a vendor account
   1. Possible consequences
      1. loss or corruption of vendor data
      2. use of storage for illicit purposes
      3. interruption of vendor data deliveries
      4. load on "u2" server (currently wain006)
2. Hacking into the vsftp server
   1. Is this likely?
3. Hacking into the lsstlnx VM
   1. Probably independent of vsftp and, therefore, no different from other VMs at SLAC

Why Existing FTP Service is Unacceptable

1. Non-anonymous (s)FTP requires a SLAC unix account and that has been deemed unacceptable by LSST project team
2. Anonymous FTP server suffers from several shortcomings:
   1. The server software cannot restart an interrupted data transfer
   2. The AFS-backed store is possibly not scalable to the hundreds of GB needed
   3. The 3-day dwell period is too risky for the data
   4. The AFS permissions combined with the 3-day swell do not allow for the type of permissions that would allow a convincing separation between the two vendor's data
   5. The dropbox does not allow for vendor management of its data at SLAC

Installation details

We are using vsftpd daemon running on a dedicated virtual machine. The machine is running a standard SLAC RHEL6 installation, with taylor and NFS access. Login is restricted to members of the sca-admin group. 

Modifications to standard installation: 

sudo yum install vsftpd
cd /etc/vsftpd
create file virtual_users.txt:
ITL
password1
e2v
password2
sudo db_load -T -t hash -f /etc/vsftpd/virtual_users.txt /etc/vsftpd/virtual_users.db

Modify standard /etc/vsftpd/vsftpd.conf as follows

12c12
< anonymous_enable=NO
---
> anonymous_enable=YES
96c96
< chroot_local_user=YES
---
> #chroot_local_user=YES
116a117
> pam_service_name=vsftpd
119,127d119
< 
< # Virtual user setup
< guest_enable=YES
< virtual_use_local_privs=YES
< pam_service_name=vsftpd_virtual
< user_sub_token=$USER
< local_root=/nfs/farm/g/lsst/u1/vendorData/ftp/$USER
< hide_ids=YES
< guest_username=lsst-ftp

Start vsftpd

sudo /etc/init.d/vsftpd restart