The proposal is to provide two or three different VMs for GlastRelease developers and users:
- Bare bones: Stripped version of Scientific Linux OS (SL5 for now; SL6 coming soon), possibly with a few extra tools installed
- GR standard: Bare bones + a recent tag of GR, externals it requires, calibration archive, moot archive and tools above and beyond those in Bare bones.
- GR stand-alone: Everything in in GR standard + whatever is required to run independent of network access to SLAC. In particular, it must have a MySQL server with current copies of calibration and moot metadata databases.
GR stand-alone won't be need right away; perhaps it will never be needed. I'll start with the other two.
Bare bones
Bare bones "hardware" configuration
This will probably only appeal to users who have a Linux host and some experience in doing system management for it. It may be preferable to GR standard for such users because of VirtualBox's "shared folder" functionality: guests can access host files so may be able to get by with a significantly smaller virtual disk. Since these users will probably be doing some tinkering anyway, I propose to package the VM with minimal memory and disk parameters. Users can adjust them once the appliance is installed on their host.
Bare bones contents
To be stripped: The basic idea is to get rid of everything not needed for code development or system management: no document processing, no graphics or sound apps, etc. All those functions exist on the host machine.
Using categories as they appear in the "Add/Remove " app I propose the following adjustments to the SL distribution:
- SL Addons: Yum utilities (convenient for managing installed packages) and, from Misc Scientific Linux Packages, only dkms (useful if kernel is upgraded)
- Desktop Environments: GNOME only (exclude KDE, Ice). Some of the accompanying optional packages are probably unnecessary.
- Applications: Uncheck all subcategories except Editors, Emacs, Graphical Internet and perhaps Text-based Internet. Within Graphical Internet, keep only Firefox and maybe Java.
- Development: Have to have subcategories "Development Libraries", "Development Tools"; maybe need "X Software Development". I think we can get along without any of the rest.
- Servers: Need "Network Servers" and probably "Server Configuration Tools". Probably can get along with the rest.
- Base System: Stick with default except maybe eliminate "Dialup Networking Support"
- Cluster Storage, Clustering, Virtualization: Don't need any of these (they come up unselected by default)
To be added: Maybe the right answer here is "nothing". I'd be inclined to add Python 2.7.x (default for RHEL5 and SL5 is awfully old: 2.4.3) and SCons 2.1.0.
Comment (Tom S.): Do we even need the Graphical Internet stuff in the VM? I could see it maybe for browsing to Confluence to fire off the installer but maybe we just add the installer? How much is a web browser inside the VM really needed? I would think that most browsing that needed to be done could be done with the host browser instead of one in the VM. (Joanne) The reason I included the browser was to make it easy to download and install arbitrary things from the Web: hard to know in advance what a developer will find useful. For Bare bones this isn't necessary since the user will surely already have shared folders but for GR Standard, especially if the host is not Linux, it could be useful. It would simplify maintenance a bit for Bare bones and GR Standard to start with the same basic SL but we could just add the browser for GR Standard, especially if the disk space is significant.
For reference, I built a 64-bit SL5 VM this morning being very aggressive in culling packages and cut out even more than what you suggested above figuring I would add in any additional packages as needed. All told, the virtual disk after installation was 2.3 GB in size uncompressed. I haven't tried compressing it yet for transfer as I wanted to actually go through and trim it down even more if I could. That will probably reduce the compressed size but not the actual disk size as they always grow and don't shrink even when you remove stuff. However, that will leave room to install things I missed without the size increasing by much.
Bare-bones VM version 1 for testing can be found here:ftp://ftp-glast.slac.stanford.edu/glast.u35/VM/SL5_32min.tar.bz2 It is 673MB to download and 2.3GB uncompressed. username: lat, password: latuser, root password: latroot
Bare-bones VM version 2 is ready for testing. It can be found at ftp://ftp-glast.slac.stanford.edu/glast.u35/VM/SL5-32min.ova . It is 874 MB in size and is 2.5 GB uncompressed. This version was exported using the 'Export Appliance' function from Virtual Box. To use it, just use the 'Import Appliance' (Ctrl+I) button in the File menu drop down.
Bare-bones VM version 3 is ready for testing. It is just version 2 with the addition of cvs and scons version 2.1.0. It can be found at ftp://ftp-glast.slac.stanford.edu/glast.u35/VM/SL5-32min-v3.ova . It is 872 MB in size and is 2.4 GB uncompressed. [The fact that it's slightly smaller than v2 even though it ostensibly has more stuff is a mystery to me (Joanne) and a bit worrisome.] Like version 2 it was exported using the 'Export Appliance' function from Virtual Box. To use it, just use the 'Import Appliance' (Ctrl+I) button in the File menu drop down.
Bare bones beta-testing
(Joanne) In order to install Guest Additions and use scons I needed the following packages:
kernel-devel
make
gcc
openmotif-devel
gcc-c++
My first attempt using scons still failed, apparently from confusion over python version. I then built and installed first python (2.7.3), then scons (2.1.0) in the VM, in /usr/local. GlastSvc built normally; I haven't tried anything else yet. python and scons to together take up 89 Mbytes.
GR Standard
GR Standard VM version 1.0.0 is ready for testing. It can be found at ftp://ftp-glast.slac.stanford.edu/glast.u35/VM/SL5-32min-GR-v1.0.0.ova . It is 2.1 GB in size, about 4 GB uncompressed. It was exported using the 'Export Appliance' function from Virtual Box. To use it, just use the 'Import Appliance' (Ctrl+I) button in the File menu drop down.
GR Standard "hardware" configuration
GR Standard should be usable without adjustments. Memory and disk sizes should be appropriate for building and running Gleam.
GR Standard proposed contents
See following section for contents of GR standard v1.0.0.
To be stripped: Use the same selections as above for Bare bones
To be added:
- Developer installation (source plus installed build products) of a current GR tag
- Associated externals (1.5 Gbyte* )
- Calibration archive * (1 Gbyte*) ?
- Moot archive (2.4 Gbyte*)
- One or both of the Installers
- GoGui (17 Mbyte as is; 10 Mbyte stripped)
- maybe RMViewer (20 Mybte as is; 13 Mbyte stripped)
* means "could and probably should be pruned"
RMViewer is not strictly necessary since in most cases it will run on the host. Besides, the SCons RM web pages handle most of RMViewer's display functions and one or the other Installer can be used for installs
Externals: Some are much larger than they need to be. For example, CLHEP 2.1.0.1 for RHEL5 32-bit is 83 Mbytes but about half of that comes from static libraries which we don't use. We could save 200 Mbyte easily, maybe significantly more, out of the 1.33 Gbyte taken up by externals for a recent GR tag.
Name |
Version |
Original size |
Pruned size |
Method |
|---|---|---|---|---|
cfitsio |
v3060A |
6.6( ) |
3.0 |
rm *.a |
CLHEP |
2.1.0.1 |
55( ) |
21 |
rm *.a |
cppunit |
1.10.2 |
11( ) |
7.5 |
trim dirs except lib, include |
extFiles |
v0r9 |
72 |
72 |
|
fftw |
3.1.2 |
2.1 |
0.0 |
rm *.a to get to 0.9 or eliminate external altogether; not used by GR |
fox |
1.4.34-gl1 |
17 |
7.8 |
trim dirs except lib, include; rm *.a; rm *.la |
gaudi |
v21r7-gl1 |
78(94) |
84 |
(remote installed external is larger than original in SLAC GLAST_EXT) |
geant4 |
9.4.p01-gl2 |
463(506) |
442 |
rm *.a |
ldf |
v07-01-01 |
4.2 |
3.5 |
exclude doc, sip and most of test |
mysql |
5.1.39 |
112(129) |
8.3 |
trim dirs except lib, include, bin. In lib save only libmysqlclient.so |
obf |
B3.0.0-gl1 |
3.2(3.7) |
3.7 |
|
OmniOrb |
4.1.4 |
66 ( > 85) |
7.4 |
trim dirs except lib, include. Delete libs we don't use. Make sym links instead of copies |
python |
2.6.5 |
140(155) |
140(155) |
|
ROOT |
v5.26.00a-gl6 |
272(288) |
263.2 (275) |
trim dirs test, tutorials. more may be possible |
swig |
1.3.31 |
4.6(5.6) |
5.6 |
|
TMineExt |
3.2.7 |
34( ) |
30 |
exclude data/TMine/examples |
xerces |
2.8.0-gl1 |
9.8(11) |
10 |
get rid of libxerces-depdom.so; we don't use it |
TOTAL |
|
1350.5 (> 1478) |
1135.8 |
|
Calibrations: Most of the 1 Gbyte calibration archive comes from CAL (838 Mbyte). I think at most half of it is possibly interesting for a typical Gleam run with real or MC data. By making some cuts on the metadata (instrument=LAT, proc_level=PROD, vend > 2008-1-1, certain flavors excluded) and throwing in some extras when too inconvenient to do otherwise, I ended up with calibration archive size = 326 Mbytes. I think this will do; it's possible a few more will be needed.
Moot:The largest contributors to the size of the moot archive are subdirectories tmp (377 Mbyte), Param (1.9 Gbyte) and Ancil (98 Mbyte). There is no reason to copy any of tmp. Most likely there is very little that Gleam needs (just a couple small top-level files and some files under Param concerning filter configuration) but this has to be confirmed. First attempt is 85 Mbytes. Need to check it's adequate.
If openafs performance is adequate we could install openafs in the VM and forget about copying in any of the calibration or moot archives.
GR standard v1.0.0
Warning
Memory and disk sizes have not been changed from bare-bones. Most likely disk at least will need to be increased. This can be done at any time after you've imported the appliance.
Warning
Guest additions was installed and working in v1.0.0 on my host machine, but it probably will not survive the export. If you want to use it, you will have to install it on your imported copy of the appliance. See instructions on the page Installing and Using a GlastRelease Appliance .
Contents
In addition to everything in bare-bones, it has GlastRelease-20-04-01, appropriate calibrations and moot archive, GoGui, RMViewer, and a user set-up file to define GLAST_EXT and so forth.
For a detailed list of contents of v1.0.0, start it up and see /usr/local/etc/SL5-32min-GR-v1.0.0.txt. Or see the same file attached here .
User environment
Both VM types will come equipped with a user account lat and of course root. For GR Standard that account will have a file glastSetup.sh which, when run, insures that scons is in path and various standard environment variables (CVSROOT, CVS_RSH, GLAST_EXT, LATCalibRoot, maybe INST_DIR) have suitable values.