Date: Fri, 29 Mar 2024 08:59:09 -0700 (PDT) Message-ID: <1111933233.8516.1711727949060@co.atl-prod-01> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_8515_840291914.1711727949059" ------=_Part_8515_840291914.1711727949059 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
You will first need a SLAC unix computer account to be access yo= ur data.
There are numerous ways to access your data. Each have their pros and co= ns which will be discussed. All data is kept on SLAC's global file system a= nd is available from all centrally managed hosts.
All experimental data is stored under /sdf/group/cryoem/exp/YYYYMM=
/YYYYMMDD-AA##
where Y is year, M is month, D is day, and AA## =
is the 4-character alphanumeric proposal number. Example: if your experimen=
t name is 20180520-CS01
, then your data will be located at
Permissions to access the data can be controlled via the elogbook <= a class=3D"external-link" href=3D"https://cryoem-logbook.slac.stanford.edu/= lgbk/experiments" rel=3D"nofollow">https://cryoem-logbook.slac.stanford.edu= /lgbk/experiments. One should add the SLAC unix ID whomever should have= access to the data to the list of 'collaborators' for the experiment.
Rsync can be used over ssh to bulk transfer/synchronise data across diff=
erent locations. The data transfer nodes (dtn01.slac.stanford.=
edu and dtn02.slac.sta=
nford.edu
have been deprecated as of January 2024. Please use centos7.slac.stanford.edu for th=
is purpose until further notice.
One can create a free Globus account at https://globusonline.org= . From there, one can download a local client (Globus Connect Personal) so that files may be copied to s= ay your laptop, or staged to another Globus Endpoint that your institution = may run.
The Globus 4 endpoint at slac#cryoem is no longer supporte=
d.
(Globus shut down all version 4 endpoints worldwide on De=
cember 18th 2023).
There is a new Globus v5 instance running but is currently (as of Feb 20= 24) only available to S3DF users. Email your microscope support contact for= an invitation to join S3DF (the new cluster), once you accept and complete= your registration at coact.slac.stanford.edu, you should be a= ble to access the slac#s3df_globus5 endpoint from globus.org= a>. You will be asked to choose an idP (identity provider), you should be a= ble to log in using your SLAC unix credentials to either option. The Cryo-E= M data is mounted on that endpoint at:
/fs/ddn/sdf/group/cryoem/exp/
If you are onsite at SLAC, you can access the data via samba/cifs. You s=
hould connect to zslaccfs
in order to browse the global d=
irectory. From there you can access the cryo-EM disks under cryo=
em
.
You should login with your SLAC Windows Account to use this.
Alternatively, you can mount it via command line.
Install cifs-utils. sudo apt-get= install cifs-utils
sudo mo= unt -t cifs -o username=3DSLAC_USERNAME,vers=3D1.0,domain=3Dslac,uid=3D`id = -u` //zslaccfs.slac.stanford.edu/cryoem/ /mnt/slac
Note that the uid
is important so that you as us=
er can have the correct permissions on your local desktop.
Alternatively, you can mount it via command line.
Install cifs-utils. sudo apt-get= install cifs-utils
sudo mo= unt -t cifs -o username=3DSLAC_USERNAME,vers=3D1.0,domain=3Dslac,uid=3D`id = -u` //zslaccfs.slac.stanford.edu/cryoem/ /mnt/slac
Note that the uid
is important so that you as us=
er can have the correct permissions on your local desktop.
You can also use the FUSE based SSHFS to mount the filesystem via ssh. I=
t is recommended to ssh into dtn02.slac.stanford.edu
in o=
rder to use this method.
Using SSHFS, the filesystem will be mounted locally so that you may brow= se the directory as you would with SAMBA etc.
BBCP provides multi-stream parallel transfers of data that allows transf= ers speeds to be significantly higher than that of scp. Details TBD.
Globus | SAMBA | SSHFS | RSYNC/SCP | BBCP | ||
---|---|---|---|---|---|---|
Software Install | Clients available, Servers should already have i= t installed | Implementation already baked into MAC OS and Win= dows. Simple install for Linux. | Requires FUSE library. Not available for Windows= (I think) | Command line tools usually already installed in = most Linux and MAC OSes. GUI's available for SCP. | Command line tools only | |
Graphical Interface | Yes (Web) | Yes (OS) | Yes (OS) | Potentially | No | |
Command line interface | Yes | Yes (standard OS) | Yes (standard OS) | Yes (standard OS) | Yes (standard OS) | |
Performance | Fast | Fast | Slow | Slow | Fast | |
Access | Anywhere | At SLAC Only | Anywhere | Anywhere | Anywhere | |
Credentials | Globus ID + SLAC Unix | SLAC Windows | SLAC Unix | SLAC Unix | SLAC Unix | |
Ease of Use | Easy | Easy | Medium | Difficult | Difficult |