IEPM-BW Installation Guide

Hardware Requirements

The monitoring host should be a machine dedicated to the IEPM-BW software. The host must have enough CPU power to run the probes, drive the MySQL database, and run the analysis. It should also have enough disk space to support the database. The recommended monitoring host hardware configuration is:

• Dual cpu xeon 3 GHz processors with hyperthreading or better
• 4 Gigabytes of memory to allow for plenty of space for MySQL and TCP buffers and workspace as well as a minimum amount of paging
• 1 Gigabit network interface, although a 10 Gigabits nic is also acceptable
• 100 Gigabytes of disk space to allow for the MySQL data base and and all the analysis reports which are generated.
• A separate disk to back up the MySql data base contents

The target host does not need to be a dedicated machine, but it should not be a busy machine.

• A gigabyte of memory is recommended
• If the machine is dedicated to being a target, 1 Gz of CPU is enough.
• Very little disk space is required. The server code is small and can reside in the home directory of the account it runs from.

Software Requirements

Following are the software requirements for the IEPM-BW monitoring node.

• Recommended operating system is the latest stable version of RedHat or any flavour of Fedora, more suitably Fedora Core 5.
• Two accounts should be created on the monitoring machine as these are needed to install and run IEPM-BW.
  • 'iepm' is a user account and is used for running IEPM-BW.
    • Account = iepm, home directory = /home/iepm
  • Second account is an account with root privileges. This is account is used for installation and administration. It's 'cyrus' in my case.
    • Account = cyrus, home directory = /home/cyrus
      You can have your own account, but don't forget to replace it at every place with your own account.
• An apache web server has to be available, and the recommended document root is '/var/www'.
• Standard MySQL must be installed on the system. Most recommended is the latest stable release, as this will facilitate assuring the upward compatibility of the code.
• Make sure that the system is configured to autostart MySQL and httpd (apache web server) upon booting. On some systems this can be done with 'chkconfig'. Commands are

        chkconfig httpd on  
        chkconfig mysqld on 

• The following /etc/sysctl.conf parameters are recommended for optimum performance

  /proc/sys/net/core/wmem_max=20971520
  /proc/sys/net/core/rmem_max=20971520
  /proc/sys/net/core/rmem_default=65536
  /proc/sys/net/core/wmem_default=65536
  /proc/sys/net/ipv4/tcp_rmem=4096 87380 20971520
  /proc/sys/net/ipv4/tcp_wmem = 4096 65536 20971520

The following ports must be open for the probes

Keep in mind to open port for each and every other service you run. e.g. HTTP, NTP etc

• To allow for running OWAMP, the monitoring and target hosts should be configured with 4 NTP servers. The OWAMP documentation discusses how to setup NTP.
• The latest version of JAVA must be installed on the monitoring host.
• Gnuplot 4.0 is required and must be installed on the monitoring host machine
• Perl should be installed locally on the machine. It can be either in /usr/bin/perl or /usr/local/bin/perl. There should be a link from where it doesn't exist to where it exists. Suppose if Perl path is /usr/bin/perl, its link should be created at /usr/loca/bin/perl. Perl modules must be installed on the system.

	 Date::Calc
	 Date::Manip
	 Date::Parse
	 Time::localtime
	 Time::Local
	 Time::Parse
	 Sys::Hostname
	 Proc::ProcessTable
	 Bundle::DBD::mysql
	 getopts.pl
	 getopt.pl
	 Getopt::Std
	 strictIO::Handle
	 File::Temp
	 Soap::Lite
	 CGI
	 Bit::Vector
	 GD

Don't' forget to set the time zone in Date::Manip configurations along with other configurations.

Important notes for later reference:

• To start the mysql daemon, use mysqld_safe&
• To set the root password after installation use: mysqladmin -u root password "new password"
• To shut mysql down, use mysqladmin -u root -p shutdown

• Decide what accounts are going to be needed. Assume the account 'cyrus' will have root privileges and be used for operations requiring root. Assume that the monitoring will run under the non-root access account 'iepm'. Both, for purposes of this documentation are assumed to be in group <account group>, iepm. For this example, I will assume that the home directory for the 'cyrus' account is /home/cyrus, and the home directory for the 'iepm' account is /home/iepm. For the sake of consistency between monitoring systems, the account 'iepm' with /home/iepm is strongly recommended.
  Download the complete package and place the source code in the directory /home/iepm/v3src
• The monitoring host has a name. This is referred to as <fully qualified monitoring host name>. e.g. iepm-ncp.ncp.edu.pk or iepm-bw.slac.stanford.edu
• The monitoring host should be given an <aliasname> that starts with 'iepm-bw'. The idea is to hide the real name of the node, but provide it with a name that can recognize its place in the world. This is important. Examples include:

  	iepm-bw.twaren.net
  	iepm-bw.cern.ch
  	iepm-bw.snv.ul-org
  	iepm-bw.fzk.de
  

• If the account mysql:mysql has not been allocated, allocate it by using the following commands:

  groupadd -g 20000 mysql
  useradd -u 20000 -g mysql mysql
  

  Normally the account is created itself along with the installation of mysql.

• Make the mysql directory '/home/iepm/mysql'.

  mkdir /home/iepm/mysql
  chown mysql:mysql /home/iepm/mysql
  

• Create a configuration file for /etc/my.cnf which MySQL will use when you install the database. Copy the following contents in to my.cnf.

  [mysqld]
  datadir=/home/iepm/mysql
  socket=/home/iepm/mysql/mysql.sock
  port=3306
  [mysqld_safe]
  datadir=/home/iepm/mysql
  socket=/home/iepm/mysql/mysql.sock
  port=3306
  [mysql.server]
  datadir=/home/iepm/mysql
  socket=/home/iepm/mysql/mysql.sock
  port=3306
  [client]
  socket=/home/iepm/mysql/mysql.sock
  port=3306
  

• If mysql is already running, it is running with the 'test' database. Kill the mysql daemons.
  Restart MySQL with mysqld_safe &. This will restart the MySQL server with the configuration file /etc/my.cnf.
• Create the data base by executing mysql_install_db.
• Change the ownership of /home/iepm/mysql by the command chown chown -R mysql:mysql /home/iepm/mysql.
• Set the database root password with the following command.

  mysqladmin -u root password <new password>
  

• You are now ready to create the directories with /home/iepm/mysql that must be owned by the iepm account. These directories are used by the iepm monitoring to track the state of the monitoring system. Execute the following commands with 'iepm:iepm' with 'iepm' replaced by the appropriate group for the 'iepm' account. i.e. it would be iepm:< youraccountgroup> in your case.

  # The data directory is where the results from the probes are stored.
  # These files are picked up by the 'load-datad' daemon and put into the database tables.
  	mkdir /home/iepm/mysql/data
  	chown iepm:iepm /home/iepm/mysql/data
  # the logs directory contain the log files from each process
  	mkdir /home/iepm/mysql/logs
  	chown iepm:iepm /home/iepm/mysql/logs
  # The pids from the daemons are stored here
  	mkdir /home/iepm/mysql/pids
  	chown iepm:iepm /home/iepm/mysql/pids
  # The files in the keepalives directory are used by the monitoring software to ascertain that
  # the daemons are running
  	mkdir /home/iepm/mysql/keepalives
  # prime the pump with:
  	touch /home/iepm/mysql/keepalives/traced.alive
  	touch /home/iepm/mysql/keepalives/pingd.alive
  	touch /home/iepm/mysql/keepalives/bw-synched.alive
  	touch /home/iepm/mysql/keepalives/pathchirpd.alive
  	touch /home/iepm/mysql/keepalives/load-datad.alive
  	touch /home/iepm/mysql/keepalives/load-scheduled.alive
  	touch /home/iepm/mysql/keepalives/owpingd.alive
  # set the permissions so that the iepm account can access them
  	chown -R iepm:iepm /home/iepm/mysql/data
  	chown -R iepm:iepm /home/iepm/mysql/logs
  	chown -R iepm:iepm /home/iepm/mysql/keepalives
  	chown -R iepm:iepm /home/iepm/mysql/pids
  

• The output from the analysis and reports must go into a directory which is writeable by the 'iepm' account. Generally this is '/home/iepm/public_html'. Make this directory with by using the commands:

  	mkdir /home/iepm/public_html
  	mkdir /home/iepm/public_html/<aliasname>
  	chown -R iepm:<account> /home/iepm/public_html
  

  Where <account> is the account your UNIX id is under. For example, for 'cyrus:iepm' <account> would be replaced by iepm.
• Create a link to this area from /var/www/html as your document root of apache will be /var/www/html.

  ln -s /home/iepm/public_html/<aliasname> /var/www/html/<aliasname>
  

• IEPM-BW requires a simple configuration file, '/etc/iepm.cnf'. Create the file by the command

  mkdir /etc/iepm.cnf
  

  Copy the following contents into the file and make the required changes.

  # required iepm configuration parameters
  # /etc/iepm.conf
  	dataBase = "iepm"
  	dataBaseHost = "<fully qualified monitoring host name>"
  	dataBaseDir = "/home/iepm/mysql"
  	iepmSrcDir = "/home/iepm/v3src"
  	updateAcct = "iepm"
  	acctGroup = "<account group>"
  	readAcct = "readonly"
  # This is provided as the monitoring host and database host could be
  # different machines.
  # It is not advisable for them to be different. In monitoring a network, 
  # one should avoid using the network for the monitoriong control and data # logging.
  	monHost = "<fully qualified monitoring host name>";
  	aliasName = "<aliasname>";
  	monHostip = "<monitoring host ip address>";
  	mysqlport = 3306;
  
  #Connie Logg 5/12/04 cal@slac.stanford.edu <- configurator's name and date please
  

  An example iepm.cnf file is below to help you in changing parameters specific to your site.

  #required iepm configuration parameters
  # /etc/iepm.conf
  
  dataBase = "iepm"
  dataBaseHost = "iepm-ncp.ncp.edu.pk"
  dataBaseDir = "/home/iepm/mysql"
  iepmSrcDir = "/home/iepm/v3src"
  updateAcct = "iepm"
  acctGroup = "iepm"
  readAcct = "readonly"
  # This is provided as the monitoring host and database host could be different machines
  # It is not advisable for them to be different. In monitoring a network, one should avoid using the
  # network for the monitoriong control and data logging.
  monHost = "iepm-ncp.ncp.edu.pk";
  aliasName = "iepm-ncp.ncp.edu.pk";
  monHostip = "202.83.167.108";
  mysqlport = 3306;
  

IEPM-BW Database Setup

A MySQL system has its own set of users which are distinct from the unix users, although the names can be the same. The recommended users are:

1. iepm - this is the account the monitoring system runs with
2. readonly - this is a readonly account for general access to the data
3. apache (or nobody) - the account used by httpd to access the database
4. cal (or other user) - the administrator account
5. root - used for MySQL system administration

I remind that these accounts are Mysql user accounts and should not be confused with system user account.

The accounts are defined by issuing grants to the account for accessing the data base.
Create a file named pws in /home/iepm/mysql i.e.

touch /home/iepm/mysql/pws

edit the file and put the password of your iepm mysql account in it.
Create the IEPM data base. Log into MySQL with the root account and password and issue the following command:

create database iepm;

The following grant commands can be set up in a text file for 'batch' entry into MySQL, or they can be entered one by one. The root account and password must be known to set up the grants. Make sure, you put the passwords in and don't use the string 'password'. Note the host name component of the grant. All these forms may not be necessary, however I have found that they are for various means of access.

#For mysql administrator ('cyrus' in this case) grant all on *.* to cyrus@localhost identified by 'password' with grant option;
grant all on *.* to cyrus@202.83.167.108 identified by 'password' with grant option;
grant all on *.* to cyrus@iepm-ncp identified by 'password' with grant option;
grant all on *.* to cyrus@'iepm-ncp.ncp.edu.pk identified by 'password' with grant option;
grant file on *.* to cyrus;

# For the iepm account used by iepm to load data into the data base:
grant all on iepm.* to iepm@localhost identified by 'password';
grant all on iepm.* to iepm@202.83.167.108 identified by 'password';
grant all on iepm.* to iepm@iepm-ncp identified by 'password';
grant all on iepm.* to iepm@'iepm-ncp.ncp.edu.pk' identified by 'password';
grant file on *.* to iepm;

# For the readonly account (report generation) - no password
grant select on iepm.* to readonly@localhost;
grant select on iepm.* to readonly@'202.83.167.108';
grant select on iepm.* to readonly@iepm-ncp;
grant select on iepm.* to readonly@'iepm-ncp.ncp.edu.pk';
grant file on *.* to readonly;

# For the web server account (may be nobody or apache or other) - no password
grant select on iepm.* to apache@localhost;
grant select on iepm.* to apache@'202.83.167.108';
grant select on iepm.* to apache@iepm-ncp;
grant select on iepm.* to apache@'iepm-ncp.ncp.edu.pk';
grant file on *.* to apache;

Creating Tables

Create the tables in iepm database. The scripts of these tables can be found here. However you can use the SQL queries in the files in create tables directory. Create tables by giving these files as input to mysql. Please check if you might need to change nodes.txt in case you have different paths for ping, grep, traceroute etc. Use the command

mysql -u root -p > file_name Database name

For example, for creating monhost table in iepm database

mysql -u root -p > monhost.txt iepm

It will prompt you for password; enter your mysql root password.

Loading the IEPM Database

Next comes the loading of data in to the database tables. Look at the following links of tables at SLAC IEPM node to have some idea of the data to be inserted in the database tables.

http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/monitoring-hosts.html
http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/NODESPECS.html
http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/targets-iepm-bw.slac.stanford.edu.html
http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/toolspecs.html
http://www.slac.stanford.edu/comp/net/iepm-bw.slac.stanford.edu/plotspecs/pltspecs-entries.html

The scripts for populating the database tables are located in v3src/mysql. You can load tables with data using CSV files. Other way you can also create your own queries, put them in a file and give as input to mysql using the command

mysql -u root -p  > file_name  databasename

At the moment, I recommend creating table by using queries.
For reference, see the following links

Populating the NODES Table
Populating the GROUP Tables
Populating the MONHOST Table
Populating the TOOLSPECS Table

For more information on the iepm-bw mysql table setup see IEPM Database Table Summary.

>cd $bandsrc
>tar -cvf v3src .tar
>gzip -c v3src.tar /tmp/v3src.tar.gz
>ls -l /tmp/v3src.tar.gz
-rw-r--r--    1 cottrell sf       125632972 May 17 12:34 /tmp/v3src.tar.gz
>rm v3src.tarcp /tmp/v3
>mv /tmp/v3src.tar.gz ./
>ls -l v3src.tar.gz
-rw-r--r--    1 cottrell sf       125632972 May 17 12:34 v3src.tar.gz
>cd /afs/slac/public/users/cottrell
>cp $bandsrc/v3src.tar.gz ./

This requires a lot of disk space (e.g. v3src.tar is ~ 180MBytes). It creates a file $bandsrc/v3src.tar.gz which is probably too big to email so it is made available by anonymous ftp. To access it use:

>ftp ftp.slac.stanford.edu
Connected to ftp.slac.stanford.edu (134.79.18.30).
220-====================================================================
220-By using this system you expressly consent to the terms and
...220 FTP server ready.
Name (ftp.slac.stanford.edu:cottrell): anonymous331 Anonymous login ok, send your complete email address as your password.
Password:
230-====================================================================
230-
230-Stanford Linear Accelerator Center Anonymous FTP Server
230-
230-====================================================================...
ftp> binary
200 Type set to I
ftp>cd users/cottrell
ftp> ls
227 Entering Passive Mode (134,79,18,30,188,42).
150 Opening ASCII mode data connection for file list
dr-xr-xr-x   2 ftp      ftp          4096 May 17 19:45 .
dr-xr-xr-x   2 ftp      ftp          4096 May 14 18:21 ..
-r--r--r--   1 ftp      ftp           335 Oct  5  2001 identity.pub
-r--r--r--   1 ftp      ftp           304 Feb  9  1996 pgp.publickey
-r--r--r--   1 ftp      ftp      125632972 May 17 19:45 v3src.tar.gz
226 Transfer complete.ftp> get v3src.tar.gz
local: v3src.tar.gz remote: v3src.tar.gz
227 Entering Passive Mode (134,79,18,30,188,44).
150 Opening BINARY mode data connection for v3src.tar.gz (125632972 bytes)
226 Transfer complete.
125632972 bytes received in 28.4 secs (4.3e+03 Kbytes/sec)
ftp>quit
>

You will need to unzip and detar the file using: tar -xzvf v3src.tar.gz 

Log onto the monitoring host node under the 'iepm' account.

Now configure the 'config' directory in v3src. The following files may need configuration:

1. The 'config' directory contains a sample .cshrc. Modify this to match the node being installed and copy it to /home/iepm/.cshrc. This will defined where java is located, and two environment variables "$bandsrc" which points to the v3src directory and "$banddata" which points to the directory where the data and reports are stored. Note that there is a link from /var/www/html to this directory.
2. cleanup-list - should be configured correctly. If there are processes which are known to hang, they should go in this file. 'bw-cleanup' uses this file to clean processes known to hang (now or in the past). If the process with the name in this list exists longer than the number of minutes indicated on its line, it will be killed.
   An example is:

   # cleanup process list
   # Connie Logg 5/18/06 cal@slac.stanford.edu
   	post-test,120
   	gnuplot,5
   	triganal,120
   	runperiod,120
   	pathload_rcv,3
   	pathchirp_rcv,3
   	/bin/ping,3
   	/bin/owping,3
   	/bin/iperf,3
   	/bin/tlaytcp,3
   	/bin/bw-iperf-client,3
   

3. owampd.conf.template, owampd.limits, and owampd.keys are configuration files for owamp. owampd.limits and owampd.keys should be ok. Look at owampd.conf.template. datadir and vardir are defined to be /home/iepm/owamp/data and /home/iepm/owamp/run respectively. If this is ok (recommended default), save owampd.conf.template as owampd.conf.
   Then create the directories:

   mkdir /home/iepm/owamp
   		mkdir /home/iepm/owamp/data
   	    mkdir /home/iepm/owamp/run
   

4. servers.alive may need to be configured. This is the file that has the information for starting the servers. It is used by keep-servers-alive to start, and if necessary, restart the servers. For a standard monitoring host, this file should be:

   bw-iperf-server,bin,-s -p 5000 -w 20M
   thrulayd,bin,
   pathload_snd,bin, -q -i
   map-updated,,
   pathchirp_snd,bin,
   owampd,bin, -c /home/iepm/v3src/config -Z >& /tmp/owampd.log &
   

The first field is the name of the script. The second field is the directory in $iepmSrcDir where the script resides. The third is a list of parameters to start it with.

1. Make copies of /etc/my.cnf and /etc/iepm.cnf in the config directory. They are used to provide configuration information to the CGI scripts.

   cp /etc/my.cnf /home/iepm/v3src/config
   	  cp /etc/iepm.cnf /home/iepm/v3src.config/iepm.cnf.<aliasname>
   

2. Install JAVA, if required. If it must be installed, the recommended installation directory is /home/iepm. To install it, copy the 'JDK...bin' in home/iepm/iepmtools to /home/iepm and execute it.
3. Configure Geoplot. http://www.slac.stanford.edu/comp/net/bw-html/v3plm.html#geoplot
4. Now assuming that database tables are loaded, run the various reports for them. report-nodes should be run first. If MONHOST is not set up correctly, none of the reports will run. You must fix MONHOST first.
5. Other reports to run are:

   report-monhost
   report-nodegroups
   report-nodespecs
   report-plotspecs
   report-table-structure
   report-toolspecs
   

6. Once all the reports run, you are ready to try the probes. 'schedule-load' will force the probes to be scheduled and run. Look in $mysqldb/data and check the data files that are output from the probes running. If they look ok, you can execute the various data loading scripts called by load-datad. These include:

   load-ping-data
   load-trace-data
   load-iperf-data
   load-tlayctp-data
   load-trace-data
   load-pathchirp-data
   load-pathload-data
   

   If all of the above tests work, and the data base is loaded with all probes, nodes, plotspecs, etc. then you can load the crontab. Crontab files are located in v3src/crontabs. You should use the one with name crontab.iepm.

   Crontab -u username  crontab.iepmbw
   

   crontab -l will give you the listing of cronjobs for the logged in account.
   This should start the system up. If there are problems, be sure to check the /home/iepm/mysql/logs directory for the output from the cronjobs.