Mobile Rack Port Forwarding

Concept

The LICOS mobile rack is isolated from SLAC Central Unix systems by a pair of firewalls, one at B33, and one in the rack itself. However, ssh port forwarding can provide the appearance that software clients running on the mobile rack are able to communicate directly with their corresponding server applications on Central Unix hosts. To achieve this connectivity, an intermediate host located between (in a network sense) the two firewalls can be used to chain together a pair of ssh forwarding tunnels for each application / protocol. In the following configuration description, the Central Unix host at the SLAC end of the tunnels is 'glast02', the intermediate machine behind the ODS-DMZ firewall is 'dcm2', and the Mobile Rack bastion host is 'lat-dmz0x'.

Additional utilities

An open-source utility called 'autossh' is used as a wrapper to invoke the ssh client processes that establish the forwarding tunnels. This utility will restart the client process if it terminates for some reason; in addition, it "piggybacks" a monitoring connection through a parallel tunnel to detect network partition conditions.

Implementation

The two firewalls are configured such that 'dcm2' can initiate outbound ssh connections towards either the mobile rack or Central Unix, but inbound connections to 'dcm2' are blocked. Therefore 'dcm2' must initiate both sets of tunnels. Accordingly, the following modifications are necessary to the configuration of 'dcm2' (which is a Taylored machine with access to AFS and NIS)

• A local home directory for the 'glastops' account is created on dcm2.
• The NIS home directory setting for 'glastops' is overridden on dcm2 by adding a record to the /etc/passwd file.
• SSH Protocol-2 RSA and DSA key-pairs are created for the glastops account on dcm2, and the public keys are added to ~/.ssh/authorized_keys2 for glastops on both glast02 and lat-dmz0x.
• A SysV-style boot-time script is installed in /etc/init.d on dcm2 to invoke the individual autossh-wrapped ssh client instances for each forwarded service.
  • The -R invocation is used to create the tunnels between dcm2 and lat-dmz0x.
  • The -L invocation is used for the tunnels between dcm2 and glast02.
  • For each service, the destination port of the -R tunnel with lat-dmz0x corresponds to the originating port of the -L tunnel with glast02.
• The option "GatewayPorts=yes" is added to /etc/ssh/sshd_config on both dcm2 and lat-dmz0x to cause the forwarding ports to bind to IN_ADDR_ANY rather than the loopback address, and thus make the forwarded ports available externally on the forwarding hosts.

The boot-time script is as follows:

#
# lattunnel     This starts and stops SSH tunnels for the LICOS mobile rack
#
# chkconfig: 345 90 10
# description: lattunnel is a boot-time script that automatically brings up \
#              an SSH tunnel to the LICOS mobile rack through which FASTCopy or \
#              MySQL transfers can occur.  It uses the "autossh" \
#              utility to maintain the tunnelled connection.
#

# set the PATH
PATH=/sbin:/bin:/usr/bin:/usr/sbin:/u/gl/glastops/Prod/bin
export PATH

# start up the tunnels
start(){
    # tunnels from this machine to Central Unix hosts
    su -s /bin/bash -c "autossh -f -M 55000 -2 -q -n -N -x -L 40000:glast02:40000 glast02" glastops
    su -s /bin/bash -c "autossh -f -M 55001 -2 -q -n -N -x -L 3306:glast03:3306 glast02" glastops
    su -s /bin/bash -c "autossh -f -M 55002 -2 -q -n -N -x -L 20022:centaurusa:22 glast02" glastops
                             
    # tunnels from the bastion hosts to this machine
    su -s /bin/bash -c "autossh -f -M 55003 -2 -q -n -N -x -R 40000:dcm2:40000 lat-dmz02" glastops
    su -s /bin/bash -c "autossh -f -M 55004 -2 -q -n -N -x -R 3307:dcm2:3306 lat-dmz02" glastops
    su -s /bin/bash -c "autossh -f -M 55005 -2 -q -n -N -x -R 20022:dcm2:20022 lat-dmz02" glastops
}

# stop the tunnels
stop(){
    killall autossh
}

# restart the tunnels
restart(){
    stop
    start
}

# show some sort of status
status(){
    ps -efwww | grep -v grep | grep autossh
}

# see how we were called
case "$1" in
    start)
        start
        ;;

    stop)
        stop
        ;;

    restart|reload)
        restart
        ;;

    status)
        status
        ;;

    *)
        echo $"Usage: %0 {start|stop|status|restart|reload}"
        exit 1
esac

The following table describes the forwarding configuration for the various applications. A graphical depiction is also available in either png or Visio format.

Application-specific notes

• FASTCopy: Since lat-dmz01:40000 is effectively redirected to glast02:40000, the fcopyd daemon that would normally listen for incoming FASTCopy transfers must be disabled on lat-dmz01. Thus it is not possible to perform a FASTCopy transfer to lat-dmz01, only from lat-dmz01 to glast02.

• MySQL: The LICOS Electronic Log (ELog) application uses a MySQL database running local to the mobile rack on lat-dmz01. This database's mysqld listens on the "standard" MySQL port 3306. Therefore, port 3307 on lat-dmz01 is redirected to glast03:3306 for access to the MOOD and FMX databases.

• CVS: Access to the LAT Flight Software CVS repository in support of the CMX build tool requires some per-user configuration in order to make use of the tunnel to centaurusa:
  • Each user must create ssh public-private key pairs for their mobile-rack account, and install their mobile-rack public keys in user@centaurusa:~/.ssh/authorized_keys and user@centaurusa:~/.ssh/authorized_keys2.
  • Each user must export CVSROOT=:ext:lat-dmz01:/nfs/slac/g/glast/flight/archive in their mobile-rack account.
  • Each user must add the following lines to user@lat-dmz01:~/.ssh/config:

    lat-dmz01*
          Port 20022
          ForwardX11 no