Page History

Design 1:

DB

Xml Schema

file-transfer.rnc

Source:

• FTMA source tar ball
• SQL Procedures to build Database for FTMA

...

• There are a number of online resources, that can help an administrator, install GridFTP. Some of them are:
  • http://www.globus.org/toolkit/docs/latest-stable/data/gridftp/admin/^{Image Removed}
• A bug in GridFTP causes intervals to be calculated incorrectly unless you use a very recent version of GridFTP or the subversion trunk version of NetLogger.
• There is generally no restriction on the version of GridFTP to be used, however, it is advised to use the latest one, with minimum security risks.

...

NetLogger forms the basis of Data collection in this service. A detailed description of installing NetLogger can be found by following this URL:http://acs.lbl.gov/NetLogger-releases/doc/trunk/manual.html#python_install^{Image Removed}
The python install section current as of Jan 26 2010, is included here for quick reference.

...

MySQLdb for MySQL

 easy_install MySQLdb 

web site: http://sourceforge.net/projects/mysql-python^{Image Removed}

psycopg2 or pgdb for PostgreSQL

 easy_install psycopg2 

web site: http://www.initd.org/pub/software/psycopg/^{Image Removed}

Install

Below are instructions for installing the Python instrumentation API and tools.

Install from source

• Unpack sources

  • Code Block
    tar xzvf netlogger-python-VERSION.tar.gz

  • Code Block
    cd netlogger-python-VERSION

• Run Python's standard install sequence

  Code Block
  python setup.py build python setup.py install

• Alternately, to install under "$NETLOGGER_HOME"

  Code Block
  export NETLOGGER_HOME=/my/path # or use setenv on csh python setup.py install --home=$NETLOGGER_HOME export PYTHONPATH=$NETLOGGER_HOME/lib/python export PATH=$PATH:$NETLOGGER_HOME/bin

...

Detailed Instructions on using NetLogger can be found here:http://acs.lbl.gov/NetLogger-releases/doc/trunk/cookbook.html^{Image Removed}

Download the Source Code as a tar ball

...

• Checkout the latest version of FTMA:
  • With username and password:

• Code Block
  svn checkout https://svn.internet2.edu/svn/perfSONAR-PS/branches/FileTransfer/

  OR
  • Anonymously:

• Code Block
  svn checkout https://anonsvn.internet2.edu/svn/perfSONAR-PS/branches/FileTransfer/

...

• Create the log Directory

  Code Block
  mkdir /var/log/perfsonar (Create the log directory as is shown by daemon_logger.conf)

• Run the SQL Procedure(mysql) on the netlogger database to produce a secondary database.

...

max_worker_lifetime      360
max_worker_processes     30
disable_echo     0
ls_registration_interval     60
ls_instance     http://192.168.117.129:9995/perfSONAR_PS/services/hLS
root_hints_url    http://www.perfsonar.net/gls.root.hints
<port 9000>
	<endpoint /perfSONAR_PS/services/FT/MA>
		service_type     MA
		module    perfSONAR_PS::Services::MA::FT
		<ftma>
			service_description    FT MA
			service_accesspoint     http://localhost:9000/perfSONAR_PS/services/FT/MA
			enable_registration     0
			service_name    perfSONAR_PS FT MA
			ls_registration_interval     60
			service_timeout     360
			query_size_limit     100
                        db_host     localhost
                        db_username     root
                        db_name     ft_ma 
                        db_password   **** 
			db_type     mysql
		</ftma>
	</endpoint>
</port>

...

The service includes a basic client tool as well.

• Code Block
  perl FTP_client.pl

This simple execution will fetch all the metadata from the service, and provide the user with a final output showing all the metadata keys mapped on source and destination ip addresses.

• Code Block

  perl FTP_client.pl --help -d Switch to debug level, one of 0,1 or 2 --debug Same as -d --url Url to the MA Service(FT) default is localhost --data Output Data as well --src source ip (string) --dst destination ip (string) --SrcPath metadata param: Source file path(string) --DestPath metadata param: Destination file path(string) --stripes metadata param: number of stripes --buffer metadata param: bufer size --block metadata param: block size --streams metadata param: number of streams --program metadata param: program used for file transfer --user metadata param: user, who requested the file transfer --initEpochTime initial Time limit in Epoch (integer) --finalEpochTime Final Time limit in Epoch (integer) --initUtcTime initial Time limit in UTC (string) --finalUtcTime final Time limit in UTC (string) --startTuple start point of results to return --tupleLimit number of tuples to return -h Print this help --help Same as -h

• Code Block
  perl FTP_client.pl -d --data --DestPath=/ --stripes=1 --src=192.168.117.128 --dst=131.225.107.12 --initEpochTime=1220000000 --finalEpochTime=1225408002

o This execution will fetch metadata where destination path, stripes, source ip address and destination ip address is defined by the parameters passed.
o The --data argument makes sure that the returned metadata is then used to fetch data with start time and end time within the given limitation.
o The -d argument provides the debug data as well.

• Code Block
  FT_client.pl -d 2 --data --stripes=1 --user=dang --startTuple=400 --tupleLimit=20