Background

We have several webservers behind the firewall that serve up scientific applications. We’ve been using a few IIS servers to act as gateway to those servers and provide services such as SSL termination.

Our IIS servers are ancient. Currently, glast-win01 and glast-win02 operate behind a load balancer, and it serves up glast-ground only. web08 is not behind a load balancer, and has no redundancy. The OSs on at least the glast-win01/glast-win02 machines is EOL and unsupported, and as such glast-win01 and glast-win02 are operating on a security exception with the assumption that we will finish the nginx migration soon. We are replacing glast-win01, glast-win02, and web08 all with the proposed service.

As for the choice of nginx, nginx as it has much better performance characteristics in the application of an HTTP reverse proxy, which is necessitated by our migration towards REST APIs which serve up scientific data for many experiments.

Configuration

Hosts

sca-www: Domain name tied to a floating IP 134.79.129.86; this is a floating IP address. sca-nginx01: Primary server, this host is a virtual machine sca-nginx02: Failover server, this host is a virtual machine scalnx12-vmm: Hypervisor for the sca-nginx01 virtual machine. scalnx13-vmm: Hypervisor for the sca-nginx02 virtual machine.

Operation

nginx is running on the two virtual machines, sca-nginx01 and sca-nginx02. Both physical machines are in the high-availability rack.

The web servers (sca-nginx01/sca-nginx02) operate in a master-failover configuration with no load balancer. Failover management is managed by keepalived, which runs on both web servers, and utilizes a floating IP address.

Software

keepalived: We use RHEL6’s “loadbalancer” child channel to install keepalived. nginx16: We use RHEL6’s Software Collections Library (SCL) channel to install a nginx16. taylor: Taylor is used to manage the hosts. See the Taylor section for more info.

Domains

Currently, our IIS servers serve up pages for several domains, including, most importantly: glast-ground.slac.stanford.edu srs.slac.stanford.edu lsst-camera.slac.stanford.edu exo-data.slac.stanford.edu cas.slac.stanford.edu

It also serves up pages for a few historical domains: www-sld.slac.stanford.edu www-bes.slac.stanford.edu pingerlod.slac.stanford.edu

There’s a few non-SLAC domains it serves as well, such as linearcollider.org and forum.linearcollider.org.

NGINX

nginx, as ran on sca-nginx01 and sca-nginx02, will operate purely in “reverse proxy” mode, forwarding requests and responses to/from an ensemble of tomcat servers. This is typically used for load balancing.

For example, a request to: glast-ground.slac.stanford.edu/DataCatalog will forward to a tomcat server running on glast-tomcat08.slac.stanford.edu on port 8080, which is behind the fiewall. A request to glast-ground.slac.stanford.edu/Pipeline-II, instead forwards to glast-tomcat09.slac.stanford.edu, also on port 8080. This is primarily to allow us to manage load on a per-application basis.

SSL/TLS

nginx will operate as an SSL termination endpoint, and we will attempt to use TLS 1.2+ for all connections across all domains, when possible, to increase security in our scientific web applications.

Taylor

On sca-nginx* machines, /etc/taylor.opts is configured as follows:

%%include opts/scalnx-v keepalived 

When including opts/scalnx-v, we also inherit the following taylor configuration:

automounter=autofs %%if ($ENV{HOSTNAME} ne 'scalnx-v03') #limit_login=u-scalnx %%endif network_device=eth0 monitoring=nagios,ganglia iptables sudo_workgroups=scalnx-vmm 

...