Content
Installation
Run server
Shell
Shell is a manual command line interface.
Connection to DB in python
from pymongo import MongoClient #client = MongoClient('localhost', 27017) client = MongoClient('psanaphi105', 27017) #, username=uname, password=pwd) db = client['calib-cxi12345'] col = db['camera-0-cxids1-0']
Connection time is 50-150ms depending on host and time.
Tentative model of the calibration store
Experiment-centric calibration data base
# Database for experiment dbexp = client["calib-cxif5315"] # Collections: col1 = dbexp["cspad-0-cxids2-0"] col2 = dbexp["cspad2x2-0-cxids2-0"] col3 = dbexp["andor-0-cxids2-0"] # Document content for dbexp doc = { "_id":ObjectId("53402597d852426020000002"), "experiment": "cxif5315" "run": 123 "detector": "cspad-0-cxids2-0" "ctype": "pedestals" "time_sec": 1516321053 "time_nsec": 123456789 "time_stamp": "2018-01-18T16:17:33.123456789-0800" "version": "v00-11-22" "facility": "LCLS2" "uid": "login-name" "host": "psanaphi102" "comments": ["very good constants", "throw them in trash immediately!"] "data_size": 32*185*388 "data_shape": (32,185,388) "data_type": "int16" "data": np.array(...) }
All meta-data information is accessible through a single-level document.
Detector-centric calibration data base
# References or DBRefs for detectors dbdet = client['calib-cspad'] col1 = dbdet['cspad-0-cxids1-0'] col2 = dbdet['cspad-0-cxids2-0'] col3 = dbdet['cspad-0-cxidsd-0'] col4 = dbdet['cspad-0-xcsendstation-0'] col5 = dbdet['cspad-0-xppgon-0'] col6 = dbdet['cspad-0-sxrbeamline-1'] col7 = dbdet['cspad-0-mectargetchamber-0'] # Document content for dbdet doc = { "_id":ObjectId("..."), "ref_id": ObjectId("534009e4d852427820000002"), etc... }
Essentially document in the detector collection has a reference to the data in the experiment collections.
Preparation of data
- Preparation of cspad data in text/unicode format for inserting takes ~1sec.
- Only limited precision data can be saved due to limit on document size 16MB.
Inserting data
Insertion time is 110-180ms.
Find data
Finding data time is 50-60us
Unpack data
Time to unpack is 350ms.
Summary
- MongoDB structure has limitations in number of levels and document size.
- server may have many DBs
- DB is a container for collections
- collection is a group of documents
- document is a JSON/BSON object of key:value pairs (dictionary). Each value may be dictionary itself etc, but further structure levels are not supported by DB structure.
- document size has hardwired limit 16MB (in 2010 increased from 4 to 16MB and devs do not want to change it). CSPAD 2Mpix*8byte(double) = 16MB, but we may expect larger detectors like Jungfrau, Epix, Andor, etc.
- Larger data size is suggested to save using GridFS; split data for chanks and save chunks in the same DB in different collections.
- JSON (text) object in MongoDB is presented in unicode...(UTF-8). Data should be converted to unicode force and back in saving retrieving.
- schema-less DB looks interesting to certain extents, but in order to find something in DB there should be a schema...
- other than that it is over-some and pretty good storage for social networks!
References
- https://docs.mongodb.com/manual/tutorial/install-mongodb-on-linux/
- http://api.mongodb.com/python/current/tutorial.html
Overview
Content Tools