This is some notes based on AFS installations of Tomcat - usually on RHEL6. They are not necessarily applicable if not using AFS.

See Also: SCA NGINX Configuration, Server Locations and Functions, https://github.com/slaclab/sca-resty (requires authorization)

Accounts and installations (on AFS)

  • ~freehep/tomcat
  • ~glast/tomcat
  • ~srs/tomcat


We usually have a scripts directory which has some useful things for starting/stopping, and the tomcat script (which start/stop wrap). 

We also have per-host environments. The tomcat app inspects hostname and will launch the appropriate tomcat instance based on that.

As such, migrating to a new version of tomcat (8.x to 8.x+1 or 9.x, etc...) usually means creating a base for it (in the appropriate BASE folder) and editing or creating a new java-env file under scripts for that version, after you've checked the tomcat script to consider newer tomcat versions.

If just upgrading a point release (8.5.100 to 8.5.103, etc...) you usually just deploy the new version on AFS.

Finding what's running

There's a few places you can inspect.

First and foremost, the sca-resty repo's main branch should represent the state of the nginx reverse proxy at all times:

https://github.com/slaclab/sca-resty/tree/main/rootfs/etc/nginx/conf.d

Config files, named per-site, are located there.


Secondly, the ~$USER/crontabs directory should represent the state of crontabs, running on RHEL 6 machines, at all times. 

Crontab files, named per-host, are defined there. Of course, if there could be skew, so you may want to verify.

You should be able to be able to find servers that are likely running with the following command:

grep "tomcat/scripts/monitor" ~$USER/crontabs/* | grep -v "~" | grep -v ":#"


scalnx-v03 - freehep, sventon, etc...

scalnx-v03 is running freehep things (under the "freehep" account) and quite a bit of other things. It also runs sventon.

Sventon seems to have issues frequently and they likely lock up the tomcat server. The core of the issue seems related to lucene, which is a search engine, and possibly the size of the svn repos.

It's not quite clear if this is all the case. Further investigation may be warranted.


/scratch partitions

Folders under /scratch partitions may fill up.

If they do, it's either because of:

  • Local Jenkins Workspaces
    • Should be safe to delete in almost all cases
  • Tomcat Logs
    • See `/scratch/$USER/tomcat(70|85)/logs`
    • Delete old common_access, probe, and wrapper logs

Proxies (nginx)

When behind an HTTP proxy (nginx, aka resty - but also apache or firewall) - server.xml MUST be configured properly.

This means server.xml needs a RemoteIpValve configured. Below is an example for nginx (configured against 4 different nginx servers).

Tomcat MUST be restarted after doing these changes.


         <Valve className="org.apache.catalina.valves.RemoteIpValve" 
              internalProxies="134.79.129.91|134.79.129.92|134.79.129.11|134.79.129.13"
              remoteIpHeader="x-forwarded-for"
              remoteIpProxiesHeader="x-forwarded-by"
              protocolHeaderHttpsValue="https"
              protocolHeader="X-Forwarded-Proto" />
  • No labels