Versions Compared

Old Version 16

changes.mady.by.user Mikhail Dubrovin

Saved on

compared with

New Version 17

changes.mady.by.user Mikhail Dubrovin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Data extraction time: 96ms. Thus returned array is "flattend" and needs to be shaped.

Interface from Murali

2018-08-03 e-mail from Murali:
I have installed Mongo 4.0 on psdb-dev. I was hoping to use their REST service but this seems to have been deprecated and eliminated since 3.6. 
So, I knocked a quick web service and have proxied it from pswww-dev. This web service (https://github.com/slaclab/psdm_mongo_ws) is a suggestion only; please let me know if you need something different. 
These are examples of getting data over HTTPS from a batch node from within cori; needless to say, the URL prefix is https://pswww-dev.slac.stanford.edu/calib_ws

Summary

  • MongoDB structure has limitations in number of levels and document size.
    • server may have many DBs
    • DB is a container for collections
    • collection is a group of documents
    • document is a JSON/BSON object of key:value pairs (dictionary). Each value may be dictionary itself etc, but further structure levels are not supported by DB structure.
      • document size has hardwired limit 16MB (in 2010 increased from 4 to 16MB and devs do not want to change it). CSPAD 2Mpix*8byte(double) = 16MB, but we may expect larger detectors like Jungfrau, Epix, Andor, etc.
      • Larger data size is suggested to save using GridFS; split data for chanks and save chunks in the same DB in different collections.
      • JSON (text) object in MongoDB is presented in unicode...(UTF-8).  Data should be converted to unicode force and back in saving retrieving.
  • schema-less DB looks interesting to certain extents, but in order to find something in DB there should be a schema...
  • GridFS works fine with document size>16GB.

...