Jenkins
We are currently running Jenkins 2.73.2 on scalnx-v06: http://srs.slac.stanford.edu/hudson
Upgrades
To upgrade the version of Jenkins:
- download the war file from http://jenkins-ci.org/
- stop the tomcat server on scalnx-v06
- copy the hudson directory and the hudson.war file in the webapps directory somewhere else, as a backup
- move the war file you downloaded to webapps/hudson.war
- restart the tomcat server
Jenkins Home
The home directory where Jenkins keeps its configuration files and build results is located at /gpfs/slac/staas/fs1/g/g.srs/services/jenkins/home
Jenkins Settings
For SRS the Jenkins settings (primarily for the slaves) are located under /nfs/slac/g/srs/hudson
Worker Nodes
There are a few worker nodes that are running experiment specific builds. The slave.jar file must be kept synchronized with the version of the hudson war file when upgrades are performed. To do this it is necessary to shut down the worker node by loggin in on the machine on which the worker node is running and killing the java process.
To find which server a worker node is running on follow: Manage Hudson -> Manage Nodes -> NODE_NAME -> System Information There you can look for HOST.
How do we run the monitor script?
Worker Node | Account | Directory | Host | Comment | |
|---|---|---|---|---|---|
| ok | exo-build01 | exodata | /nfs/slac/g/exo/software/hudson/exo-build01 | scalnx01 | crontab ok from ~exodata/crontabs/scalnx01 |
exo-build02 | exodata | ~/hudson | www.exo200.org | ||
| ok | exo-build-srcf-01 | exodata | /home/exodata/jenkins | SRCF (slurm) | This agent is started on-demand via ssh |
| ok | exo-build-rhel6-64k | exodata | /nfs/slac/g/exo/software/hudson/exo-build-rhel6-64k | rhel6-64j | Runs rhel6-64. This agent is started on-demand via ssh |
| ok | srs-build01 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/srs-build01 | scalnx-v06 | cronjob ok from ~srs/crontabs/scalnx-v06 |
| ok | srs-build02 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/srs-build02 | scalnx-v02 | crontab ok from ~srs/crontabs/scalnx-v02 |
| ok | lsst-build01 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/lsst-build01 | scalnx01 | cronjob ok from ~srs/crontabs/scalnx01. Running java at 32 bits. |
| ok | lsst-build02 | srs | /gpfs/slac/staas/fs1/g/g.srs/hudson/lsst-build02 | scalnx-v02 | crontab ok from ~srs/crontabs/scalnx-v02 |
| ok | freehep-build01 | freehep | /nfs/slac/g/jas/hudson/freehep-build01 | scalnx01 | cronjob ok from ~freehep/crontabs/scalnx01 |
| ok | fermi-build01 | glast | /nfs/slac/g/glast/ground/software/hudson/ | fermilnx-v03 | cronjob ok from ~glast/crontabs/fermilnx-v03 |
| ok | hps-build01 | lcdprod | /gpfs/slac/staas/fs1/g/g.lcd.mc/prj/sw/hudson/hps-build01 | scalnx-v06 | no crontab |
| ok | lcsim-build01 | lcdprod | /gpfs/slac/staas/fs1/g/g.lcd.mc/prj/sw/hudson/lcsim-build01 | scalnx-v02 | no crontab |
cta-build01 | ctadata | /nfs/slac/g/agis/repo/hudson/rhel5-64/ | scalnx01 | Temporarily off since scalnx01 was updated to Rhel6 | |
| fermi-ppa-pc90719 | glastrm | /Users/glastrm/jenkins | ppa-pc90719 | This is now started using launchd from ~glastrm/Library/LaunchDaemons/monitor-jenkins.plist, but does not start automatically after a reboot. Instructions to start: ssh glastrm@ppa-pc90719 (password in escrow as macglastrm) | |
fermi-win04 | glast | d:\jenkins | glast-win04 | This has been configured to run as a windows service using the instructions here: | |
fermi-bldlnx06 | glastrm | /u/gl/glastrm/hudson/bldlnx06 | bldlnx06 | ||
fermi-bldlnx11 | glastrm | /u/gl/glastrm/hudson/bldlnx11 | bldlnx11 | ||
fermi-bldlnx12 | glastrm | /u/gl/glastrm/hudson/bldlnx12 | bldlnx12 | ||
cdms-build-rhel6-64d | cdmsdata | /u/dm/cdmsdata/jenkins | rhel6-64d |
Launching Jobs
Most jobs are configured to start on SCM changes by polling the repository with scheduled tasks. The SCM polling has been causing problems, so we have migrated to a different strategy: triggering the jobs remotely upon SCM changes.
SVN repositories
For the FreeHEP and the SRS repositories we have installed post-commit hooks to notify Jenkins of SCM changes. Jenkins will the internally figure out which jobs to start.
The scripts are installed in the following locations:
SRS: /nfs/slac/g/srs/subversion/hooks
Freehep: /nfs/slac/g/jas/svnbackup/svnrepositories/svn/hooks
CVS repositories
CVS hooks work differently than SVN hooks. SVN hooks are triggered at different stages of the commit, while CVS hooks are triggered for each commit. For Fermi we modified the file /nfs/slac/g/glast/ground/javacvs/CVSROOT/loginfo (has to be checked out, don't modify it on the file system directly). Each time there is a commit we execute the script /nfs/slac/g/glast/ground/jenkins/post-commit-hook which touches a file in /nfs/slac/g/glast/ground/jenkins/jobs with the name of the module that was modified. We have then installed a cronjob on fermilnx-v07 that runs the script /nfs/slac/g/glast/ground/jenkins/launch_jenkins_builds that is responsible to launching the corresponding jobs in Jenkins.