Installing the Conditions Database Locally

Overview

The HPS conditions database contains time-dependent detector information for use by reconstruction and analysis software.  The primary database is hosted at JLAB and is periodically replicated to SLAC.  The JLAB database is accessible only internally within jlab.org whereas the SLAC one is public and can be accessed from anywhere over the internet as long as the port is not blocked for outbound traffic. 

Should you want to run the HPS software without an internet connection available, then this database needs to be installed locally.  These instructions will cover how to install MySQL on your machine, configure a database and the accounts for HPS, replicate the database to your machine, and finally how to configure the software to use this local copy of the database.

Setting Up MySQL

The exact details of installing the MySQL client and server will not be covered in great depth, but complete instructions can be found in the MySQL documentation itself, e.g. MySQL Linux Installation.

For instance, on Redhat using yum, you would start by installing the necessary packages.

sudo yum install mysql-server mysql

The server should then be started up.

service mysqld start

You will need to login to the MySQL console client as the root user.

mysql -u root

And then the root password should be set for localhost.

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('new-password');
mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('new-password');

The root account is like the Unix "superuser" with complete privileges on all databases, so take care as to how this account is configured for security.  In general, you should prefer to use other accounts for your local work, but root will be used in these instructions for operations such as creating new databases.

Creating the Conditions Database and User Account

From the root account, you should create the (initially empty) database for the conditions system from the console after the initial installation.

mysql> create database hps_conditions;

You should also create a read-only account that will be used to run jobs, as root should not be used for this!

mysql> CREATE USER 'hpsuser'@'localhost' IDENTIFIED BY 'mypassword';
mysql> GRANT SELECT ON hps_conditions.* TO 'hpsuser'@'localhost';

The restricted hpsuser account will be used to run local reconstruction or analysis jobs, but it will not have any privileges to change the database.

If you intend to run the reconstruction offline (or cannot have online access to jlab database on hpsdb.jlab.org), you also need to create the hps_run_db database (v2 is the most updated version), and grant access to it to your mysql user:

mysql> create database hps_run_db_v2;
mysql> GRANT SELECT ON hps_run_db_v2.* TO 'hpsuser'@'localhost';

Creating a Database Dump

You must acquire a SQL dump from the JLAB database which will be loaded locally.

You will need to be logged into a JLAB machine like jlabl3 for this step, and you must also have a MySQL account with the correct privileges. 

mysqldump -h hpsdb.jlab.org -ujeremym -pXXXXXXXX --disable-lock-tables hps_conditions &> jlab_conditions_db.sql

The -u argument should have the actual database user account (mine is used here as an example) and the -p argument must specify the password.

The file jlab_conditions_db.sql will contain a complete copy of the table structure and all the data in SQL format.

If you do not have an account on the JLAB database and do not wish to have one created for you, then you must ask one of the HPS database administrators to create this file for you.

You can also dump the database from your local machine (if you have an internet connection), using the following command:

mysqldump -f --lock-tables=false -h hpsdb.jlab.org -u hpsuser -p hps_conditions > hps_conditions_db.sql

Your local copy, however, will not be complete as the hpsuser does not own all the necessary select privileges; the -f flag allows to skip the problematic tables. The reduced dumpfile works for most of the reconstruction needs: however, if you need some tables (the "view" ones) which are missing because of this privileges issue, you have to ask one of the DB administrator to provide a dump for you. The password for the hpsuser account on hpsdb.jlab.org may be found hardcoded in the jlab_connection.prop file (located in conditions/src/main/resources/org/hps/conditions/config).

If you intend to run the reconstruction without any network resource, you also need a local dump of the hps_run_db_v2 database:

mysqldump --lock-tables=false -h hpsdb.jlab.org -h hpsuser -p hps_run_db_v2 > hps_run_db_v2.sql

beware that this file is huge and the dump operation may require several minutes.

Loading the SQL File

Next, assuming you have transferred the SQL file to your local computer, you can use the command line tool to load it into a local database.

mysql -D hps_conditions -h localhost -P 3306 -u root -pXXXXXXXX < jlab_conditions_db.backup.sql

Now the JLAB database should be installed on your computer. Use the same syntax if you also need the hps_run_db_v2 database loaded locally on your machine.

To check the content of the loaded databases you can issue the command:

mysqlshow -u root -p hps_run_db_v2

in this case you need to provide the root password you set upon mysql configuration ath the beginning of the procedure.

Using the Local Database

The local conditions database will not be used by default so you need to create a config file, e.g. local.prop, with the connection parameters in it.

user: hpsuser
password: XXXXXXXX
database: hps_conditions
hostname: localhost

If you plan not to use any network resource to run the reconstruction, before running it it is necessary to change the location of the hps_run_db_v2 database your machine points to, to prevent it to search for it over the network (on the hpsdb.jlab.org machine). To do so, you need to edit your RunManager.java file (located in run-database/src/main/java/org/hps/run/database) and fix the DEFAULT_CONNECTION_PARAMETERS instance, in which the name of the database and its location are hardcoded. Change the hpsdb.jlab.org machine name to your localhost (127.0.0.1), and arrange accordingly the username, its password and/or the database name, should this be needed to macth with your mysql initial configuration.

Should you have an internet connection available, you need not make any change as the hps_run_db_v2 database is read from the hpsdb.jlab.org machine by default (as it is hardcoded in hps-java). The hps_conditions database is, on the other hand, read locally through the config file local.prop, which can be located anywhere on your computer.

Finally, a Java system property needs to be set whenever you run any of the command line tools for HPS on your machine.

java -Dorg.hps.conditions.connection.file=/path/to/local.prop -cp hps-distribution-bin.jar [className] [args]

or if using a runnable jar

java -Dorg.hps.conditions.connection.file=local.prop -jar hps-distribution-bin.jar [args]

In the log messages printed out from the job, you should see something like the following.

Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: opening connection to jdbc:mysql://mysql-node03.slac.stanford.edu:3306/
Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: host mysql-node03.slac.stanford.edu
Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: port 3306
Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: user rd_hps_cond_ro
Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: database rd_hps_cond
Thu Jan 22 13:18:33 PST 2015 :: DatabaseConditionsManager :: INFO :: connection opened

But the actual settings printed there should match the ones for your local database, e.g. the host should be localhost and not the SLAC or JLAB host.

Guidelines for Usage

In general, you should prefer not to make a alterations or additions to the local database.  These procedures are instead designed for running reconstruction and analysis without an internet connection on your local machine.  Any official changes to be used in production, such as recon passes run at JLAB, must be made on the JLAB database through the HPS database administrators or other users with the correct privileges.  In order for others to use any conditions you might like to add to the system, the best approach will be adding that information to the primary database.