Page History

...

There are a number of online resources, that can help an administrator, install GridFTP. Some of them are:
- *http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/*^{Image Modified}
A bug in GridFTP causes intervals to be calculated incorrectly unless you use a very recent version of GridFTP or the subversion trunk version of NetLogger.
There is generally no restriction on the version of GridFTP to be used, however, it is advised to use the latest one, with minimum security risks.

...

The following Python modules may be needed by the NetLogger pipeline to interact with the database. To install these modules, either use a package manager such as Debian's APT, the RedHat/etc. yum, FreeBSD ports, etc., use Python's easy_install command from setuptools or download and install from source. The easy_install command and download URL are given below.

MySQLdb for MySQL

Code Block
easy_install MySQLdb

web site: http://sourceforge.net/projects/mysql-python

psycopg2 or pgdb for PostgreSQL

Code Block
easy_install psycopg2

web site: http://www.initd.org/pub/software/psycopg/

...

Install from source

Unpack sources

Code Block
tar xzvf netlogger-python-VERSION.tar.gz

Code Block
cd netlogger-python-VERSION

Run Python's standard install sequence

Code Block
python setup.py build


python setup.py install

Alternately, to install under "$NETLOGGER_HOME"

Code Block
export NETLOGGER_HOME=/my/path # or use setenv on csh


python setup.py install --home=$NETLOGGER_HOME

*


export PYTHONPATH=$NETLOGGER_HOME/lib/python

**


export PATH=$PATH:$NETLOGGER_HOME/bin

*

Patching NetLogger to hadle GridFTP logs(buggy/nob-buggy):

...

Download the Source Code as a tar ball

Code Block
tar \-xvf FileTransfer_MA.1b.tar
* mkdir /var/log/perfsonar *
Run the SQL Procedure(mysql) on the netlogger database to produce a secondary database. This secondary database is the one, the FTMA service will be interacting with.

...

Checkout the latest version of FTMA:
- With username and password:

Code Block
svn checkout https://svn.internet2.edu/svn/perfSONAR-PS/branches/FileTransfer/

^{Image Removed} *

OR

Anonymously:

Code Block
svn checkout https://anonsvn.internet2.edu/svn/perfSONAR-PS/branches/FileTransfer/

^{Image Removed} *

Configuration

Code Block

*

mkdir /var/log/perfsonar

*

 (Create the log directory as is shown by daemon_logger.conf)

Run the SQL Procedure(mysql) on the netlogger database to produce a secondary database. This secondary database is the one, the FTMA service will be interacting with.

Settings for bin/daemon.conf file

Code Block

title	bin/daemon.conf
borderStyle	solid


max_worker_

...

lifetime&nbsp;&nbsp;&nbsp;&nbsp; 60
max_worker_

...

processes&nbsp;&nbsp;&nbsp;&nbsp; 30
disable_echo&nbsp;&nbsp;&nbsp;&nbsp; 0
ls_registration_interval&nbsp;&nbsp;&nbsp;&nbsp; 60
ls_instance&nbsp;&nbsp;&nbsp;&nbsp; [http://192.168.117.129:9995/perfSONAR_PS/services/hLS

...

]
root_hints_

...

url&nbsp;&nbsp;&nbsp; [http://www.perfsonar.net/gls.root.hints

...

]
<port

...

 9000>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <endpoint /perfSONAR_PS/services/FT/MA>

...


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; service_type&nbsp;&nbsp;&nbsp;&nbsp; MA
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; module&nbsp;&nbsp;&nbsp; perfSONAR_PS::Services::MA::FT

...


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <ftma>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; service_description&nbsp;&nbsp;&nbsp; FT MA
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; service_accesspoint&nbsp;&nbsp;&nbsp;&nbsp; [http://localhost:9000/perfSONAR_PS/services/FT/MA

...

]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; enable_registration&nbsp;&nbsp;&nbsp;&nbsp; 0
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; service_name&nbsp;&nbsp;&nbsp; perfSONAR_PS FT MA
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ls_registration_interval&nbsp;&nbsp;&nbsp;&nbsp; 60
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; service_timeout&nbsp;&nbsp;&nbsp;&nbsp; 60
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; query_size_limit&nbsp;&nbsp;&nbsp;&nbsp; 100000
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; db_host&nbsp;&nbsp;&nbsp;&nbsp; <database host>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; db_username&nbsp;&nbsp;&nbsp;&nbsp; <database user>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; db_name&nbsp;&nbsp;&nbsp;&nbsp; <database name>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; db_password&nbsp;&nbsp; <database password>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; db_type&nbsp;&nbsp;&nbsp;&nbsp; <database type>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </ftma>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </endpoint>
</port>

How to run the Service:

Code Block
* cd bin

...


* sh FtpMaExecute.sh \--skip-input \|\| \--help

- Attempts to install the missing, perl modules.
- Stops any previous running instance of the service.
- Creates a backup of any existing log file.
- Everytime the script executes, it copies the log file from the main log directory to an hourlyfolder in the same directory. The folder is named as: Ftp_MM-DD-YY:HH
- Starts a new Instance of the service by running daemon.pl

...

The service includes a client tool to do some preliminary testing. This tool is present inside the bin dir as well.

How to run the Service:

Code Block
* sh FtpMaExecute.sh --skip-input \|\| --help

o Attempts to install the missing, perl modules.
o Stops any previous running instance of the service.
o Creates a backup of any existing log file.
o Everytime the script executes, it copies the log file from the main log directory to an hourlyfolder in the same directory. The folder is named as: Ftp_MM-DD-YY:HH
o Starts a new Instance of the service by running daemon.pl

Client Application:

The service includes a basic client tool as well.

Code Block
perl FTP_client.pl

This simple execution will fetch all the metadata from the service, and provide the user with a final output showing all the metadata keys mapped on source and destination ip addresses.

Code Block
perl FTP_client.pl --help

-d Switch to debug level, one of 0,1 or 2
--debug Same as -d
--url Url to the MA Service(FT)
default is localhost
--data Output Data as well
--src source ip (string)
--dst destination ip (string)
--SrcPath metadata param: Source file path(string)
--DestPath metadata param: Destination file path(string)
--stripes metadata param: number of stripes
--buffer metadata param: bufer size
--block metadata param: block size
--streams metadata param: number of streams
--program metadata param: program used for file transfer
--user metadata param: user, who requested the file transfer
--initEpochTime initial Time limit in Epoch (integer)
--finalEpochTime Final Time limit in Epoch (integer)
--initUtcTime initial Time limit in UTC (string)
--finalUtcTime final Time limit in UTC (string)
--startTuple start point of results to return
--tupleLimit number of tuples to return
-h Print this help
--help Same as -h

Code Block
perl FTP_client.pl -d --data --DestPath=/ --stripes=1 --src=192.168.117.128 --dst=131.225.107.12 --initEpochTime=1220000000 --finalEpochTime=1225408002

...

o This execution will fetch metadata where destination path, stripes, source ip address and destination ip address is defined by the parameters passed.
o The --data argument makes sure that the returned metadata is then used to fetch data with start time and end time within the given limitation.
o The -d argument provides the debug data as well.

Code Block
FT_client.pl -d 2 --data --stripes=1 --user=dang --startTuple=400 --tupleLimit=20

Child pages

Versions Compared

Old Version 22

New Version 23

Key

MySQLdb for MySQL

psycopg2 or pgdb for PostgreSQL

Install from source

Patching NetLogger to hadle GridFTP logs(buggy/nob-buggy):

Download the Source Code as a tar ball

Configuration

Settings for bin/daemon.conf file

How to run the Service: