You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 3
Next »
Page about kubernetes clusters ad-build and ad-build-dev
Runs on hardware in s3df
Allocatable:
cpu: 64
ephemeral-storage: 152933498761
hugepages-1Gi: 0
hugepages-2Mi: 2816Mi
memory: 259679512Ki
pods: 220
System Info:
Machine ID: 92faa81e90af4e65ba73d3007e42519e
System UUID: ce9ba000-5727-11ed-8000-3cecefd8e38e
Boot ID: 96386228-b4ab-4836-b764-b22d4dfc0cda
Kernel Version: 4.18.0-372.32.1.el8_6.x86_64
OS Image: Red Hat Enterprise Linux 8.6 (Ootpa)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.31
Kubelet Version: v1.28.8
Kube-Proxy Version: v1.28.8
How to access
ad-build-dev cluster: https://k8s.slac.stanford.edu/ad-build-dev
ad-build cluster: https://k8s.slac.stanford.edu/ad-build
- After following the commands in those links, you will need to request access to the cluster from Claudio.
- Recommended to install k9s tool https://k9scli.io/topics/install/
Other Notes
- Kubernetes cluster is intended to be just for build system.
- Can use local machine or nodes in the kubernetes cluster to create docker images (Don't need access to s3df/afs filesystem if all modules/dependencies are uploaded to GitHub, as they should be)
- Docker can be ran on local machine, but not on s3df, it is intended to use apptainer instead, if build enviornment wants to be passed around. So when this build system is finished, developers/users won't use docker directly, instead will use apptainer and pull docker images from artifact storage (if needed)
Current
- Get the build system container running on the kluster, see if you can use the actions/actions-runner-controller: Kubernetes controller for GitHub Actions self-hosted runners
- Lets do it vanilla first (running build system container)
- Create the image using base image: Package actions-runner (github.com)
- TODO: push the docker image to a registry so anyone can pull it
Dockerfile (Here temporarily, these are the only 2 files you need to get this to work)
# base
FROM ubuntu:22.04
# set the github runner version
ARG RUNNER_VERSION="2.316.0"
# update the base packages and add a non-sudo user
RUN apt-get update -y && apt-get upgrade -y && useradd -m docker
# install python and the packages the your code depends on along with jq so we can parse JSON
# add additional packages as necessary
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
curl jq build-essential libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip
# cd into the user directory, download and unzip the github actions runner
RUN cd /home/docker && mkdir actions-runner && cd actions-runner \
&& curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
&& tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
# install some additional dependencies
RUN chown -R docker ~docker && /home/docker/actions-runner/bin/installdependencies.sh
# copy over the start.sh script
COPY start.sh start.sh
# make the script executable
RUN chmod +x start.sh
# since the config and run script for actions are not allowed to be run by root,
# set the user to "docker" so all subsequent commands are run as the docker user
USER docker
# set the entrypoint to the start.sh script
ENTRYPOINT ["./start.sh"]
start.sh
#!/bin/bash
ORGANIZATION=$ORGANIZATION
ACCESS_TOKEN=$ACCESS_TOKEN
# Generate organization registration token
REG_TOKEN=$(curl -L \
-X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/orgs/${ORGANIZATION}/actions/runners/registration-token | jq .token --raw-output)
cd /home/docker/actions-runner
./config.sh --url https://github.com/${ORGANIZATION} --token ${REG_TOKEN}
cleanup() {
echo "Removing runner..."
./config.sh remove --unattended --token ${REG_TOKEN}
}
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
./run.sh & wait $!
- do 'docker image ls' to ensure its there
- Then you must be an organization administrator, and make a personal access token with the "admin:org" and "repo" scope to create a registration token for an organization (REST API endpoints for self-hosted runners - GitHub Docs)
- Copy the token, and use it in the next step
Run the docker image
docker run \
--env ORGANIZATION=<ORG> \
--env ACCESS_TOKEN=<PERSONAL-TOKEN> \
--name runner1 \
runner-image
Replace <ORG> with the organization name
Replace <PERSONAL-TOKEN> with the token you created above
- And now your runner should be registered and running
- When done testing make sure to 'ctrl+c' and 'stop' and 'remove' the container
- Start the image using kubectl for our ad-build kubernetes cluster you created above
- For now, use a local registry
- kubectl create deployment runner1 --image=runner-image
- Then we can use that for building buildroot. One of the workflows will be it checking out on /scratch/ in s3df, then build, and output results there.
Resource (Slightly altered its files): How to containerize a GitHub Actions self-hosted runner | by Alessandro Baccini | Medium
Deployment of an image (running container) ex: Using kubectl to Create a Deployment | Kubernetes
pnispero@PC100942:~$ kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
deployment.apps/kubernetes-bootcamp created
pnispero@PC100942:~$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
kubernetes-bootcamp 1/1 1 1 6s
pnispero@PC100942:~$ kubectl delete deployment kubernetes-bootcamp
deployment.apps "kubernetes-bootcamp" deleted
pnispero@PC100942:~$ kubectl get deployments
No resources found in default namespace.
pnispero@PC100942:~$