Where is my Cryo-EM Data?

Prerequisites

• You will need an active SLAC unix account to access your data (contact Lisa Dunn).
• You will need to register as an S3DF user in the "CryoEM" facility with your SLAC unix username. Please go to coact.slac.stanford.edu, then go to Repos→Request Facility Access and select "CryoEM", then submit. For further information, see: https://s3df.slac.stanford.edu/public/doc/#/accounts-and-access?id=access

All experimental data is kept on the S3DF network filesystem and is available from the following hosts:

• S3DF interactive pool (see: https://s3df.slac.stanford.edu/public/doc/#/interactive-compute?id=interactive-pools). Note that logging into an S3DF bastion host (e.g. ssh s3dflogin.slac.stanford.edu ) does not automatically log you into an interactive node, the experimental and group data will not be available from login hosts.
• S3DF DTN hosts (see: https://s3df.slac.stanford.edu/public/doc/#/data-transfer?id=data-transfer)
• S3DF Globus endpoint via the #s3df_globus5 data collection (see: https://s3df.slac.stanford.edu/public/doc/#/data-transfer?id=globus)

NOTE: The SDF cluster has been decommissioned as of August 2024. All Cryo-EM experimental and group project data is accessible from the S3DF cluster.

Data Transfer Methods

rsync/scp/bbcp

The unix command line utilities rsync , scp , and bbcp (https://www.slac.stanford.edu/~abh/bbcp/bbcp.pdf) can be used over ssh to bulk transfer/synchronise data across different locations. The data transfer nodes (dtn01.slac.stanford.edu, dtn02.slac.stanford.edu, and the centos7.slac.stanford.edu cluster have been decommissioned. Please log into the load-balanced hostname s3dfdtn.slac.stanford.edu with your SLAC unix credentials instead). You will need to be registered as a CryoEM facility user in S3DF before you can access the S3DF DTN hosts (see Prerequisites above). Once logged into an S3DF DTN host, the data will be mounted at:

• Experimental data: /fs/ddn/sdf/group/cryoem/exp/<YYYYMM>/<YYYYMMDD>-<proposal>_<instrument>
• Group project data: /fs/ddn/sdf/group/cryoem/g/<proposal>

Globus

One can create a free Globus account at https://globusonline.org. If your institution is a part of the Globus network, you can register with your institution's credentials. From there, download the client (Globus Connect Personal) so that files may be copied a local host, or to another Globus Endpoint run by your institution, if available. You may also use the Globus web client (https://www.globus.org/).

The Globus 4 endpoint at slac#cryoem is no longer supported.(Globus shut down all version 4 endpoints worldwide on December 18th 2023).

You will need to be registered as a CryoEM facility user in S3DF before you can access the S3DF Globus endpoint (see Prerequisites above). Once your S3DF registration has been approved, you can log into Globus with your home institution's credentials (if they are a part of the Globus network) or a personal Globus account and access the #s3df_globus5 data collection. You will be asked to authenticate with your SLAC unix credentials. Once authenticated, the data will be mounted at the following paths:

• Experimental data: /fs/ddn/sdf/group/cryoem/exp/<YYYYMM>/<YYYYMMDD>-<proposal>_<instrument>
• Group project data: /fs/ddn/sdf/group/cryoem/g/<proposal>

SAMBA

If you are onsite at SLAC, you can access the data via samba/cifs. You should connect to zslaccfs in order to browse the global directory. From there you can access the cryo-EM disks under cryoem.

You should login with your SLAC Windows Account to use this.

1. On your Linux machine, open a terminal window.
2. Install the necessary software with the command sudo apt-get install -y samba samba-common python-glade2 system-config-samba.
3. Type your sudo password and hit Enter.
4. Allow the installation to complete.
5. Open a new file browser window.
6. At the bottom of the left navigation pane, click "Other Locations"
7. At the bottom of the window, in the "Connect to Server" field, Type smb://zslaccfs/
8. Open the cryoem directory.
9. Log in with your SLAC Windows account username and password, leave the "Workgroup" field as the default value or if using Ubuntu the domain should be "slac".
10. You can now browse the CryoEM disks in your Linux file browser.

Alternatively, you can mount it via command line. 

1. Install cifs-utils. sudo apt-get install cifs-utils

2. Create a directory where you want to mount it. sudo mkdir /mnt/slac

3. sudo mount -t cifs -o username=SLAC_USERNAME,vers=1.0,domain=slac,uid=`id -u` //zslaccfs.slac.stanford.edu/cryoem/ /mnt/slac

   Note that the uid is important so that you as user can have the correct permissions on your local desktop.

Alternatively, you can mount it via command line. 

1. Install cifs-utils. sudo apt-get install cifs-utils

2. Create a directory where you want to mount it. sudo mkdir /mnt/slac

3. sudo mount -t cifs -o username=SLAC_USERNAME,vers=1.0,domain=slac,uid=`id -u` //zslaccfs.slac.stanford.edu/cryoem/ /mnt/slac

   Note that the uid is important so that you as user can have the correct permissions on your local desktop.

SSHFS

You can also use the FUSE based SSHFS to mount the filesystem via ssh. It is recommended to ssh into s3dfdtn.slac.stanford.edu in order to use this method. Please note you will need to be a registered S3DF user (see Prerequisites above) to access the S3DF DDTN hosts.

Using SSHFS, the filesystem will be mounted locally so that you may browse the directory as you would with SAMBA etc.

Summary