NewAr k8s Jenkins¶

Archived (pre-2022)

Preserved for reference only -- likely outdated. View original | Last updated: May 2020

All new architecture Jenkinses fully managed "as code".

To achieve this we use:

GIT to store all configuration
Jenkins Job DSL Plugin(job-dsl-plugin (Github)) is used to create/configure Jobs in Jenkins instances
Jenkins Configuration as Code Plugin(configuration-as-code-plugin (Github)) to configure Jenkins instances(credentials, security, plugins)
Jenkins helm chart(jenkins (Github)) to provision Jenkins instance, install plugins

Initial Jenkins provision and upgrade/reconfiguration with Helm/JCasC¶

New Jenkins instance provision and further its re/configuration is performed by Helm and Configuration as Code Plugin(JCasC)

The flow is following:

First configuration is checked out from GIT repository as helm chart. All configuration is included in helm chart including JCasC part which lays under Master.JCasC namespace in values.yaml.
During second step helm template is compiled with vars from values.yaml and secrets.yaml. Sensitive data like tokens, ssh keys etc. is kept in secrets.yaml file and encrypted with SOPS(sops (Github)) before storing in GIT. Helm automatically decrypts secrets.yaml during templetisation. Compiled helm chart then provisoned into Kubernetes creating corresponding resources for Jenkins instance(Deployment, Service, Persistent volume etc.). Jenkins instance itself gets its configuration via JCasC plugin. Thus after successful helm run we receive Jenkins instance with following:
desired set of modules installed
enabled and configured security(AuthN/AuthZ)
desired set of credentials
configured Jenkins shared libraries
custom tools installation configuration
cloud plugin configuration
customised Jenkins parameters like JavaOpts for example
environment variables like JAC_ENVIRONMENT and JAC_HOST later used by DSL and regular Jobs to identify on which Jenkins instance they are ran
initial set of Jobs(in our case it's Init Job that then provisions Ceed Jobs for each project/integration)

Ceed Jobs are special in a way that no real work performed by these instead Ceed Jobs execute Job DSL code to create/configure regular Jobs in Jenkins.

Each project that wants to have its jobs in Jenkins must have Ceed Job that then downloads DSL code from project's GIT repo and create Jobs, views and other things related to this project. Ceed Jobs for each project/integration provisioned by special Init Job that bootstrapped under Master.JCasC.ConfigScripts.initial-jobs namespace in values.yaml:

Master:
  JCasC:
    ConfigScripts:
      initial-jobs: |
        jobs:
          - script: >
              job('bln-jenkins-init-job') {
                triggers {
                    scm 'H/5 * * * *'
                }
                scm {
                  git {
                    branch('origin/master')
                    remote {
                      url('git@bitbucket.org:inneractive-ondemand/bln-jenkins-configuration.git')
                      credentials('bitbucket_jenkins_deploy_key')
                    }
                  }
                }
                steps {
                  dsl {
                    external "src/dsl/*.groovy"
                    removeAction("DELETE")
                    removeViewAction("DELETE")
                  }
                }
              }

Init Job has it's code and configuration stored in GIT. Thus to add new project/integration into Jenkins the following config file needs to be adjusted(changes are made via PR): projects-config.groovy (Bitbucket)

The following is a snippet from this config file:

environments {
    fybercore {
        production {
            projectsToAutomate = [
                [projectName: "bln-k8s-common-helmfile", url: "git@bitbucket.org:inneractive-ondemand/bln-k8s-common-helm.git", scriptPath: "automation/dsl/Jenkinsfile", credentials: "bitbucket_jenkins_deploy_key"],
                [projectName: "mamba", url: "git@bitbucket.org:inneractive-ondemand/mamba.git", scriptPath: "config/jenkins/dsl/Jenkinsfile", credentials: "bitbucket_jenkins_deploy_key", branch: "DEVOPSBLN-1278"],               
            ]
        }
        staging {}
        all {}
    }
    inneractive {
        fairbidproduction {
            projectsToAutomate = [
                [projectName: "mamba", url: "git@bitbucket.org:inneractive-ondemand/mamba.git", scriptPath: "config/jenkins/dsl/Jenkinsfile", credentials: "bitbucket_jenkins_deploy_key", branch: "DEVOPSBLN-1278"],
                [projectName: "boa-blackadder", url: "git@bitbucket.org:inneractive-ondemand/blackadder.git", scriptPath: "config/jenkins/dsl/Jenkinsfile", credentials: "bitbucket_jenkins_deploy_key", branch: "bln_boa_load_tests"],                
            ]
        }
    }
}

So after this step we have fully configured Jenkins instance but with no Jobs that can perform any work for us(build, test, deploy etc.).

Ceed Jobs run¶

On first step a Ceed Job checks out DSL code from its repository. Each project's Ceed Job is pointed to corresponding project's GIT repo, thus allowing to manage project's jobs/views independently from others.
A Ceed Job is basically just pipeline job that pointed to specific Jenkinsfile inside project's GIT repo, this is shown in the code snippet above. Thus in that particular example pipeline script will be taken from automation/dsl/Jenkinsfilein git@github.com:SponsorPay/aws-infrastructure-code.git. The content of the Jenkinsfile is the following:

node {
    checkout scm

    jobDsl  targets: [
                'automation/dsl/jobs/*.groovy'
            ].join('\\n'),
            removedJobAction: 'DELETE',
            removedViewAction: 'DELETE',
            lookupStrategy: 'SEED_JOB'

    deleteDir()

}

The script, as can be seen, takes all groovy files from automation/dsl/jobs and executes them with Job DSL plugin creating/changing/deleting jobs, views and/or other resources in Jenkins.

From this point Jenkins has regular jobs to perform real work(build, test, deploy etc.).