Design:
http://confluence.slac.stanford.edu/display/IEPM/FTMA+Design
Source:
Pre-Req:
You might need to install the following perl modules if not already installed. Easiest way to do this is to use cpan e.g. "cpan module_name"
Perl:
- Module::Load
- HTTP::Daemon
- XML::SAX
- Config::General
- aliased
- Readonly
- Term::ReadKey
- DBI::DBD
- DBD::mysql
- DBD::SQLite
- Class::Accessor
- Class::Fields
- Params::Validate
- Statistics::Descriptive
- Data::UUID
- IO::Interface
- DateTime
- Error
- Date::Manip (this may require perl upgrade to 5.10 or above, directions)
- Log::Dispatch::FileRotate
- Log::Log4perl
Additionally more components may be required, which may come up when one tries to run the startup script of the service (section "How to run the service"):
- XML::LibXML
Installation:
Theoretically, the service can work with any kind of File Transfer protocol. For test purposes and limiting the large amount of possibilities we use GridFTP as a primary File Transfer Tool.
...
- There are a number of online resources, that can help an administrator, install GridFTP. Some of them are:
- A bug in GridFTP causes intervals to be calculated incorrectly unless you use a very recent version of GridFTP or the subversion trunk version of NetLogger.
- There is generally no restriction on the version of GridFTP to be used, however, it is advised to use the latest one, with minimum security risks.
NetLogger Installation
NetLogger forms the basis of Data collection in this service. A detailed description of installing NetLogger can be found by following this URL:http://acsconfluence.slac.lblstanford.govedu/NetLogger-releasesdisplay/doc/trunk/manual.html#python_install
The python install section current as of Jan 26 2010, is included here for quick reference.
Install Python Version
Prerequisites
The following Python modules may be needed by the NetLogger pipeline to interact with the database. To install these modules, either use a package manager such as Debian's APT, the RedHat/etc. yum, FreeBSD ports, etc., use Python's easy_install command from setuptools or download and install from source. The easy_install command and download URL are given below.
...
MySQLdb for MySQL
...
Code Block |
---|
easy_install MySQLdb |
web site: http://sourceforge.net/projects/mysql-python
...
psycopg2 or pgdb for PostgreSQL
...
Code Block |
---|
easy_install psycopg2 |
web site: http://www.initd.org/pub/software/psycopg/
Install
Below are instructions for installing the Python instrumentation API and tools.
...
Install from source
...
- Unpack sources
Code Block tar xzvf netlogger-python-VERSION.tar.gz
Code Block cd netlogger-python-VERSION
- Run Python's standard install sequence
Code Block python setup.py build python setup.py install
- Alternately, to install under "$NETLOGGER_HOME"
Code Block export NETLOGGER_HOME=/my/path # or use setenv on csh python setup.py install --home=$NETLOGGER_HOME export PYTHONPATH=$NETLOGGER_HOME/lib/python export PATH=$PATH:$NETLOGGER_HOME/bin
Patching NetLogger to handle GridFTP logs(buggy/nob-buggy):
To patch your NetLogger distribution, you could just drop this file into the download directory under python/netlogger/parsers/modules/ , then reinstall (e.g. python setup.py install).
Executing NetLogger
Detailed Instructions on using NetLogger can be found here:http://acs.lbl.gov/NetLogger-releases/doc/trunk/cookbook.html
PerfSONAR_PS FTMA
Download the Source Code as a tar ball
...
- Create the log Directory
Code Block mkdir /var/log/perfsonar (Create the log directory as is shown by daemon_logger.conf)
- Create a mysql database for FTMA service to interact with:
Code Block mysql -u $USER -p $PASS -e "create database ft_ma"
- Run the SQL Procedure(mysql) on the netlogger database to produce a secondary database.
...
Settings for bin/daemon.conf file
Change "db_name", "db_username" and "db_password" in the following file:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
max_worker_lifetime 360 max_worker_processes 30 disable_echo 0 ls_registration_interval 60 ls_instance http://192.168.117.129localhost:9995/perfSONAR_PS/services/hLS root_hints_url http://www.perfsonar.net/gls.root.hints <port 9000> <endpoint /perfSONAR_PS/services/FT/MA> service_type MA module perfSONAR_PS::Services::MA::FT <ftma> service_description FT MA service_accesspoint http://localhost:9000/perfSONAR_PS/services/FT/MA enable_registration 0 service_name perfSONAR_PS FT MA ls_registration_interval 60 service_timeout 360 query_size_limit 100 db_host localhost db_username root db_name ft_ma db_password **** db_type mysql </ftma> </endpoint> </port> |
How to run the Service:
Code Block |
---|
* cd bin
* sh FtpMaExecute.sh \--skip-input \|\| \--help
|
- Attempts to install the missing, perl modules.
- Stops any previous running instance of the service.
- Creates a backup of any existing log file.
- Everytime the script executes, it copies the log file from the main log directory to an hourlyfolder in the same directory. The folder is named as: Ftp_MM-DD-YY:HH
- Starts a new Instance of the service by running daemon.pl
Client Application:
The service includes a client tool to do some preliminary testing. This tool is present inside the bin dir as well.
How to run the Service:
Before running the service, we must have the database. So far we only created the database and did not copy any tables/contents. To do this we must have a file that performs sql functions of adding and filling tables. The file should be a part of the FTMA package retrieved via svn checkout and should be placed inside "FileTransfer/perfSONAR_PS-FileTransfer/contrib" directory. The file can used in the following manner (you might need to do sudo):
If mysql not started then start it:
Code Block |
---|
/etc/init.d/mysqld start
|
Load the mysql dump into a database:
Code Block |
---|
mysql -D ft_ma < mysql_backup
|
Once done do the following to start the service:
Code Block |
---|
* cd contrib
* sh FT_MA_Startup.sh \ |
Code Block |
* sh FtpMaExecute.sh --skip-input || \--help |
...
- Attempts to install the missing, perl modules.
...
- Stops any previous running instance of the service.
...
- Creates a backup of any existing log file.
...
- Everytime the script executes, it copies the log file from the main log directory to an
...
- hourlyfolder in the same directory. The folder is named as: Ftp_MM-DD-YY:HH
...
- Starts a new Instance of the service by running daemon.pl
Client Application:
The service includes a basic client tool as wellto do some preliminary testing. This tool is present inside the bin dir.
Code Block perl FTPFT_client.pl
This simple execution will fetch all the metadata from the service, and provide the user with a final output showing all the metadata keys mapped on source and destination ip addresses.
...