...
Jenkins
We are currently have two separate versions of Hudson running for:
The SRS version just got updated to the latest pre-2.0 version: 1.398 (see all versions here).
This particular version had to be patched due to a bug in running maven processes on remote nodes. We followed the instructions on how to upgrade the version of sisu-guice 3.0.1.
Goals
The main goal is to have only one Hudson server running. It will handle builds from different experiments delegating the execution to worker nodes.
To do this we need to:
- Re-organize the EXO slaves; they are currently pretty messy.
- These are Tony's notes:
Code Block The Hudson slave code for EXO is in /nfs/slac/g/exo/software/hudson/ It contains all the jar files, directories and scripts for exo-build03. In it there is also a directory for exo-build01. The idea was to create a directory for exo-build03, parallel to the exo-build01 one. The problem is that many EXO users rely on these directories being where they are. The main thing that would have to stay the same is the /nfs/slac/g/exo/software/hudson/builds directory, which is itself already a symlink to builds-rhel5 (the rhel4 stuff is obsolete and can be deleted). However inside the hudson task itself there may be other assumptions about file location, built into the script which runs the job. - For these reasons we decided to leave this as it is for the moment.
- These are Tony's notes:
- Find a server to run the LSST builds
- Import in SRS the Fermi builds.
Distribution
The Hudson war file is located at: /u/ey/freehep/hudson/. There is a directory for each version, containing hudson.war. There is also a current symlink that is used to load the most recent version in the server.xml on the server in which hudson is running.
Hudson Home
running Jenkins 2.73.2 on scalnx-v06: http://srs.slac.stanford.edu/hudson
Upgrades
To upgrade the version of Jenkins:
- download the war file from http://jenkins-ci.org/
- stop the tomcat server on scalnx-v06
- copy the hudson directory and the hudson.war file in the webapps directory somewhere else, as a backup
- move the war file you downloaded to webapps/hudson.war
- restart the tomcat server
Jenkins Home
The home directory where Jenkins The home directory where Hudson keeps its configuration files and build results is located at tomcat/BASE/HOST/home/.home on the HOST where it is running.
...
/gpfs/slac/staas/fs1/g/g.srs/services/jenkins/home
Jenkins Settings
For SRS the Hudson settings Jenkins settings (primarily for the slaves) are located under /nfs/slac/g/srs/hudson
Worker Nodes
There are a few worker nodes that are running experiment specific builds. The slave.jar file must be kept synchronized with the version of the hudson war file when upgrades are performed. To do this it is necessary to shut down the worker node by loggin in on the machine on which the worker node is running and killing the java process.
...
Worker Node | Account | Directory | Host | Comment | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ok | exo-build01 | exodata | /nfs/slac/g/exo/software/hudson/exo-build01 | scalnx01 | crontab ok from ~exodata/crontabs/scalnx01 | |||||
exo-build02 | exodata | ~/hudson | www.exo200.org | |||||||
| ok | exo-build-srcf-01 | exodata | /home/exodata/jenkins | SRCF (slurm) | This agent is started on-demand via ssh | |||||
| ok | exo-build-rhel6- | yakut1464k | exodata | /nfs/slac/g/exo/software/hudson/ | exo-build-rhel6-64k | rhel6-64j | Runs rhel6-64. | yakut14 | This agent is started on-demand via ssh | |
| ok | srs-build01 | srs | /nfsgpfs/slac/staas/fs1/g/g.srs/hudson/srs-build01 | scalnx01 | cronjob scalnx-v06 | cronjob ok from ~srs/crontabs/scalnx-v06 | ||||
| ok | srs-build02 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/srs-build02 | scalnx-v02 | crontab ok from ~srs/crontabs/scalnx-v02 | |||||
| ok | lsst-build01 | srs | /nfsgpfs/slac/staas/fs1/g/g.srs/hudson/lsst-build01 build01 | scalnx01 | cronjob ok from ~srs/crontabs/scalnx01. Running java at 32 bits. | |||||
| ok | lsst-build02 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/lsst-build02 | scalnx-v02 | crontab ok from ~srs/crontabs/scalnx-v02 | |||||
| ok | freehep-build01 | freehep | /nfs/slac/g/jas/hudson/freehep-build01 | scalnx01 |
| lcsimcronjob ok from ~freehep/crontabs/scalnx01 | ||||
| ok | fermi-build01 | glast | /nfs/slac/g/glast/ground/software/hudson/ | fermilnx-v03 | cronjob ok from ~glast/crontabs/fermilnx-v03 | |||||
| ok | hps-build01 | lcdprod | / | nfsgpfs/slac/staas/fs1/g/g.lcd | /.mc/prj/sw/hudson | lcddata01 |
| /hps-build01 | scalnx-v06 | no crontab |
| ok | lcsim | fermi-build01 | glastlcdprod | / | nfsgpfs/slac/staas/fs1/g/ | glast/ground/softwareg.lcd.mc/prj/sw/hudson/ | glastlnx20 | lcsim-build01 | scalnx-v02 | no crontab |
cta-build01 | ctadata | /nfs/slac/g/agis/repo/hudson/rhel5-64/ | scalnx01 | Temporarily off since scalnx01 was updated to Rhel6 | ||||||
| fermi-ppa-pc90719 | glastrm | /Users/glastrm/jenkins | ppa-pc90719 | This is now started using launchd from ~glastrm/Library/LaunchDaemons/monitor-jenkins.plist, but does not start automatically after a reboot. Instructions to start: ssh glastrm@ppa-pc90719 (password in escrow as macglastrm) | ||||||
fermi-win04 | glast | d:\jenkins | glast-win04 | This has been configured to run as a windows service using the instructions here: | ||||||
fermi-bldlnx06 fermi-bldmac01 | glastrm | /u/gl/glastrm/jenkins/bldmac01 | bldmac01 | hudson/bldlnx06 | bldlnx06 | |||||
fermi-bldlnx11 | glastrm | /u/gl/glastrm/hudson/bldlnx11 | bldlnx11 | |||||||
fermi-bldmac02 bldlnx12 | glastrm | /u/gl/glastrm/hudson/bldlnx12 | bldlnx12 | |||||||
cdms-build-rhel6-64d | cdmsdata | /u/dm/cdmsdata/jenkins | /bldmac02bldmac02 | rhel6-64d |
Launching Jobs
Most jobs are configured to start on SCM changes by polling the repository with scheduled tasks. The SCM polling has been causing problems, so we have migrated to a different strategy: triggering the jobs remotely upon SCM changes.
SVN repositories
For the FreeHEP and the SRS repositories we have installed post-commit hooks to notify Jenkins of SCM changes. Jenkins will the internally figure out which jobs to start.
The scripts are installed in the following locations:
SRS: /nfs/slac/g/srs/subversion/hooks
Freehep: /nfs/slac/g/jas/svnbackup/svnrepositories/svn/hooks
CVS repositories
CVS hooks work differently than SVN hooks. SVN hooks are triggered at different stages of the commit, while CVS hooks are triggered for each commit. For Fermi we modified the file /nfs/slac/g/glast/ground/javacvs/CVSROOT/loginfo (has to be checked out, don't modify it on the file system directly). Each time there is a commit we execute the script /nfs/slac/g/glast/ground/jenkins/post-commit-hook which touches a file in /nfs/slac/g/glast/ground/jenkins/jobs with the name of the module that was modified. We have then installed a cronjob on fermilnx-v07 that runs the script /nfs/slac/g/glast/ground/jenkins/launch_jenkins_builds that is responsible to launching the corresponding jobs in Jenkins.