1. Build Workflow example
    1. TODO: Make a flow chart ppt when done creating
    2. Overall plan: runner checks out repo to /sdf/group/ad/eed/ad-build/<user>/ → runners request to backend build cluster to start new build container -> builder pod starts a new container on our ad-build cluster -> write log to database
    3. Steps broken down:

      1. Runners request to backend build cluster, uses a rest api to send a POST request to 'build' endpoint with data like this:

        curl -X POST \
  'https://accel-webapp-dev.slac.stanford.edu/api/cbs/v1/component/build' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
	"organization": "str",
	"componentName": "str",
	"branchName": "str",
	"actionUser": "str"
}'
      2. Backend builder pod talks to component database, grabs the build environment based off the organization, componentName and branchName

      3. Backend builder pod then starts the build environment using build-deployment.yml with parameters (componentName, branchName, imageName, actionUser, buildInstructions)

        1. Use Declarative Management of Kubernetes Objects Using Kustomize | Kubernetes to configure the build-job.yml for the component
        2. build-job.yml will be a job with a volumeMount to s3df-dev-container-ad-group, and template parameters to label the container (componentName-branchName), and provide image name, and command to start build (buildInstructions)
      4. backend builder pod then logs to branches_in_development.
      5. backend builder pod sends the name of the build container thats running, back to runner,
      6. runners last job is to print the filepath to where the container is being built, and othe useful info like 'status'
      7. if triggered by action (push to main/ pull request), then we want to provide the Report to user which can include
        1. filepath to user of the build output
        2. status (successful built / fail).
        3. time duration
      8. LAST STEP: have the build container push its artifact somewhere where a test container can download the artifact then do its testing.

Backend meeting with Claudio/Jerry/Patrick 5-28-24

  1. The backend should do most everything, capture status, log, start build, etc.
  2. talked about api version, 
  3. creating the new branch, do we want just the cli, or what if they use just regular git. We will still think about this. CLI can checkout using the url of the component repo
  4. User should be able to build remotely (using ad-build) or build locally in a container
  5. CLI should have 'completed' command to merge to main
  6. create a tag should be associated with the version that a 
  7. dev needs a tool starting from now, this version associate this branch. Problem is when you create a new tag, when the user update an old branch or old version, you associate the tag that belong to that branch. We have to manage different versions of the same time.
    1. have a  main that always point to most recent version
    2. then have development branch that points to respective version
    3. version automatically managed by backend
  8. We should have external/internal components. We should have a way to checkout external component and build, but not tracked by build system. CLI should be similar for both
    1. ex: Epics component should be external component, 
  9. we need to carve up space for user 
  10. every user can have environment variables for github username, s3df username, and we can pass to the backend rest api if needed
  11. todo: come up with http header from cli to backend and fill out his excel sheet for cbs-api. 
    when they create repo or create new branch, they need authorization, which is the information you (may) pass to backend rest api, that can authorize. 
    TODO: Add claudio to ad-build-test org
  12. claudio wants github to store the docker images for now, but long-term goal we want a docker registry on s3df

core-build-system notes

  1. Example: Adding fields to the component schema, as well as the api. What classes to change:
    1. Rest API:
      1. controller/ComponentController.java → @RestController() 
        1. This is where you define the api commands
          1. @PostMapping, @GetMapping, @DeleteMapping
      2. mapper/ComponentMapper.java → @Mapper
        1. This is only made when you have a field that has fields of its own, and you need to map it to main schema
    2. Service:
      1. This is the backend logic that the rest api commands call
      2. service/ComponentService.java → @Service()
        1. ex: create() - check no conflict, check dependencies, then create the component and save to mongodb, then return the created component id back to user
    3. Repository:
      1. This is backend logic that extends MongoRepository for functions regarding the repository like 'boolean existsByName(string name)"
        1. repository/ComponentRepository.java
    4. Schema:
      1. model/Component.java
      2. dto/ComponentDTO.java → @Schema()
        1. This is where you define the db document schema
      3. dto/newComponentDTO.java →@Schema()
        1. What's the difference between componentDTO and newComponentDTO?
    5. Exception Handling:
      1. exception/ComponentAlreadyExists → @ResponseStatus
      2. exception/ComponentNotFound→ @ResponseStatus
    6. Testing:
      1. test/....../controller/ComponentControllerTest.java
      2. test/....../service/ComponentServiceTest.java

Proof of concept development

Used configMap to pass in information from backend to build container. So build container can echo the build parameters passed in

  • Commands:
    • kubectl create configmap build-config --from-file=build-request.
      properties
    • kubectl describe configmaps build-config
    • kubectl create -f build-config.yml

todo:  Work on test project like test-ioc, with bom. claudio can use this to test

  • we can make the lsit of components just have boost and epics for now
  • Then we want to get to the point where a pod is executed and start_build.py can read the bom of the ioc

todo: Work on getting testing flow designed

  • Unit tests firsts, the ones you can run right after the build
  • may want a specific directory name that the container test script can run
  • Come up with backend api for it, which just records to db i think, 
  • We can assume basic/unit tests are automatically ran if a build is started

Flow:

  1. User command CLI $ bs run build  which does the build flow.
    1. Pass to backend the component, branch, and user headers
    2. backend looks into component DB for the development image
    3. backend mounts /mnt  at /sdf/scratch/ad/ad-build/ for downloading src code
    4. backend mounts /build  at /sdf/groups/eed/ad/ad-build/ for the build scripts
    5. backend mounts configMap at /config/build_config.json for build request information
    6. starts the build container and calls start_build.py 
    7. start_build.py then performs a build and outputs its results at the top of repo directory
  2. start_build.py  will then call start_test.py
  3. start_test.py will then look into directories in the src code called /test/unit_tests/ and run those


Group meeting 6-6-24

Group Meeting 6-6-24 - LCLSControls - SLAC Confluence (stanford.edu)



  • No labels