Overview
Two commercial vendors competing for LSST business have been awarded contracts for a set of preliminary devices. Before shipment, each device will undergo a series of tests by the vendor, producing ~25 GB of data. These data must be transferred to SLAC, analyzed, archived and distributed to other LSST laboratories prior to a "pre-ship review". Only after successfully passing this review will a device be authorized for shipment. It is expected that SLAC will receive multiple data deliveries per month although they are not precisely scheduled and deliveries may be bundled.
Requirements
- Vendors must be able to control the timing of the data delivery.
- The intranet within the company as well as their connection with the internet can be overwhelmed by these deliveries
- Vendors insist on controlling the exact timing for the electronic data transfers
- Vendors must be able to restartan aborted data delivery.
- The internet connection between SLAC and the two vendors is subject to intermittent instability.
- The internet connection to at least one of the vendors is quite slow (100 Mbps)
- The need to restart a large and time-consuming delivery from scratch would cause an unacceptable delay
- Vendors must be able to create, modify, or delete files in their FTP areas
- The transfer buffer must be able to hold multiple data deliveries per vendor, so at least 200 GB
- LSST must do its best to prevent data from Vendor A from being visible to Vendor B, and vice versa
Proposed Solution
- LSST operated advanced FTP service
- vsftp server software: very secure, high performance, restartable transfers, private ftp-only accounts
- installed and running on LSST service VM (VM is "SCS Standard")
- New FTP server is configured to have ownership privs on a single NFS partition: /nfs/farm/g/lsst/u2 (which will be a short-term buffer from which a permanent archive will be made)
- Individual vsftp accounts for Vendors A and B.
Potential Security Issues and Mitigations
- Hacking into a vendor account
- Possible consequences
- loss or corruption of vendor data
- use of storage for illicit purposes
- interruption of vendor data deliveries
- load on "u2" server (currently wain006)
- Possible consequences
- Hacking into the vsftp server
- Is this likely?
- Hacking into the lsstlnx VM
- Probably independent of vsftp and, therefore, no different from other VMs at SLAC
Why Existing FTP Service is Unacceptable
- Non-anonymous FTP requires a SLAC unix account and that has been deemed unacceptable by LSST project team
- Anonymous FTP server suffers from several shortcomings:
- The server software cannot restart an interrupted data transfer
- The AFS-backed store is possibly not scalable to the hundreds of GB needed
- The 3-day dwell period is too risky for the data
- The AFS permissions combined with the 3-day swell do not allow for the type of permissions that would allow a separation between the two vendor's data
- The dropbox does not allow for vendor management of its data at SLAC
Installation details
We are using vsftpd daemon running on a dedicated virtual machine. The machine is running a standard SLAC RHEL6 installation, with taylor and NFS access. Login is restricted to members of the sca-admin group.
Modifications to standard installation:
sudo yum install vsftpd
cd /etc/vsftpd
create file virtual_users.txt:
ITL
password1
e2v
password2
sudo db_load -T -t hash -f /etc/vsftpd/virtual_users.txt /etc/vsftpd/virtual_users.db
Modify standard /etc/vsftpd/vsftpd.conf as follows
12c12
< anonymous_enable=NO
---
> anonymous_enable=YES
96c96
< chroot_local_user=YES
---
> #chroot_local_user=YES
116a117
> pam_service_name=vsftpd
119,127d119
<
< # Virtual user setup
< guest_enable=YES
< virtual_use_local_privs=YES
< pam_service_name=vsftpd_virtual
< user_sub_token=$USER
< local_root=/nfs/farm/g/lsst/u1/vendorData/ftp/$USER
< hide_ids=YES
< guest_username=lsst-ftp
Start vsftpd
sudo /etc/init.d/vsftpd restart