Chef Configuration Management

Table of Contents:

Introduction

Chef is a configuration management tool (like Puppet, Ansible, SaltStack, CFEngine). It is a tool which manages the configuration of centrally managed Linux servers, compute clusters, and desktops at SLAC. Examples of configuration items Chef manages include: sudo privileges, login access privileges, logging, software repositories, cronjobs, baseline security configuration. Chef is the configuration management tool for CentOS 7 and later, Red Hat Enterprise Linux (RHEL) 7 and later, and Ubuntu 16.04 and later. Operating systems earlier than those (RHEL 5 and 6, Solaris) are centrally managed using Taylor (a locally written configuration management tool).

Installing Chef

To get Chef installed on a SLAC owned Linux server, contact unix-admin@slac.stanford.edu . To get Chef installed on a SLAC owned Linux desktop, contact ithelp@slac.stanford.edu .

If you prefer to install Chef yourself, that is also possible.  Run this command as root (or sudo):

 curl -s yum.slac.stanford.edu/go-chef | sudo -i /bin/sh

You can also place the above command in your kickstart %post script if you are doing automated network installations. If you want to use a non-default chef configuration, you can create a json file named /root/kickstart-chef.json with your configuration options and it will be used by the go-chef script.  You can email unix-admin@slac.stanford.edu for help with this.

Chef Architecture

• Chef Server - the Chef Server has the cookbook code which is run on each client, and information about each client manged by Chef which is updated after every chef-client run. Each host which runs chef contacts the Chef Server before the client run to get any updated code, and after each client run, to report success or failure, and provide a full report of the updated status based on the most recent run.
• Chef Client - there is a chef-client cronjob on each host managed by Chef which runs the chef-client code on a routine basis.
• Chef Automate Server - the Chef Automate server is an Operations Dashboard which provides details about everything managed by Chef, including status and error messages. The chef-client run on each host sends information to the Chef Automate Server, so the various dashboards provide OCIO with the current state of all centrally managed hosts.
• GitHub Organization - the central repository for Chef code is maintained in a SLAC OCIO GitHub organization: https://github.com/SLAC-CHEF/
• Jenkins Server - all Chef code goes through a Continuous Integration / Continuous Delivery Pipeline before being automatically delivered to the Chef Server. Automated tests using Test Kitchen are run on Vagrant VMs which are spun up by Jenkins. If all tests pass, and an OCIO system administrator presses the "Proceed" button, then updated code is delivered to the Chef Server.

Chef log files and directories

Chef-client logs are sent to syslog and a local log file. You can view the logs using these methods on each host which is managed:

sudo less -r /var/log/chef/client.log
sudo journalctl -t chef-client
sudo grep -w chef-client /var/log/everything

The logs are also sent to the central syslog server, and to Splunk.  On the central syslog server, the log can be viewed here (this is for OCIO staff only):

ssh loghost
grep -w chef-client /u2/today/SYSLOG/daemon

Directories with Chef information, and some useful Chef commands:

Common configuration options

(this is a work in progress....)

Steps to provide an OCIO admin full access to Chef

These are the steps to grant an OCIO admin full access to the Chef Infrastructure.
Note: for minimal access (eg, to create GitHub Pull Requests and GitHub Issues), just steps 1a and 1b are needed.
Everyone in unix-admin should have steps 1a and 1b done, and optionally all steps for full access.

1. GitHub: Grant access to https://github.com/orgs/SLAC-CHEF organization
   1. Log into github as the admin for the SLAC-CHEF organization
      1. escrow -c systems display slac-chef-admin
   2. Invite the person using their github account on this page
      1. https://github.com/orgs/SLAC-CHEF/people
   3. The default permission level for members of SLAC-CHEF is read-only
      1. this is so we can invite any SLAC user to be a member so they can collaborate with us
      2. add the unix-admin user to the scs-unix team on this page so they have write access to repos
         
         1. https://github.com/orgs/SLAC-CHEF/teams/scs-unix/members
   4. The unix-admin person should create an ssh key pair for github use if necessary
      1. https://github.com/settings/keys
2. Chef Server: create account on Chef Server
   1. Log into https://chef01.slac.stanford.edu
   2. ...
3. Automate Server: create account on Chef Automate 2 Server
   1. Log into https://chef-automate2.slac.stanford.edu
   2. ...
4. Jenkins Server: create account on Jenkins Server
   1. escrow -c systems display chef-jenkins
   2. Log into http://chef-build01.slac.stanford.edu:8080
   3. ...
5. Provide instructions on how to do the following:
   1. clone an existing cookbook, make a change, push changes to github, press the "approve" button on Jenkins
   2. create a new cookbook, with Jenkinsfile
      
      1. new github repo, with scs-unix team write access
      2. new jenkins project
      3. verify cookbook permissions on chef server (write access for jenkins user only)
      4. add cookbook to a role, if appropriate
   3. how to test changes on cookbook before pushing to production
   4. how to bypass Jenkins pipeline for emergencies (eg, Jenkins server is down)