Speeding up spot based Jenkins jobs¶

What is the issue¶

As we use Spot.io to run tasks in Jenkins via Kubernetes agent, sometimes you might witness delays on startup of job execution, which is usually caused by either of the following:

1. Pod's resource request is not configured or configured incorrectly¶

In this case node allocated by Spot cannot guarantee enough resources to run the task full steam.

Please note!

Regardless what level of performance you expect from Jenkins, you should always setup resources.requests!

2. Jenkins is waiting for a node to be allocated¶

Here Jenkins waits for a new node to be selected from the pool and launched.

How do we solve it¶

In both scenarios we are going to update pod definition yaml file¶

1. Setup correct resources.requests¶

What's going on¶

We can specify how much resources a Container needs by appending adding resources.requests field to your pod definition file/code-block.

Scheduler uses this information to decide which node to place the Pod on, also kubelet reserves at least this amount of that system resource specifically for that container to use.

Bit of theory JFYI → Manage Resources Containers

Example¶

Let's pretend you have a Jenkins pipeline, described in ${YOUR_REPO}/Jenkinsfile, which builds an image of your app with kaniko.

To implement the above approach you should add resources → requests → memory/cpu on the container level, hence in spec → containers → $name

kind: Pod
apiVersion: v1
metadata:
  name: myPod
spec:
  containers:
  - name: node
    image: nginx:latest
    imagePullPolicy: Always
    tty: true
    resources:
      requests:
        memory: 1024Mi
        cpu: 300m

2. Setup nodeSelector and tollerations to use "pre-warmed" nodes¶

What's going on¶

DevOps team has configured few special pre-warmed nodes (headroom in Spot language), which are explicitly reserved and ready to host specifically configured pods.

You can assign those, by appending tolerations and nodeSelector fields to your pod definition file/code-block.

Bit of theory JFYI → Taint And Toleration

Example¶

Let's reuse the same example. You still have this ${YOUR_REPO}/Jenkinsfile, which builds an image of your app with kaniko. Now you need to make it "suitable" for pre-warmed nodes.

For that you should add this particular part to your pod description file/code-block under the spec field

  tolerations:
  - key: "noschedule"
    operator: "Equal"
    value: "jenkins-headroom"
    effect: "NoSchedule"
  - key: "noexecute"
    operator: "Equal"
    value: "jenkins-headroom"
    effect: "NoExecute"
  nodeSelector:
    service: jenkins-headroom

So in the end it will look something like

kind: Pod
apiVersion: v1
metadata:
  name: myPod
spec:
  containers:
  - name: node
    image: nginx:latest
    imagePullPolicy: Always
    tty: true
    resources:
      requests:
        memory: 1024Mi
        cpu: 300m
  tolerations:
  - key: "noschedule"
    operator: "Equal"
    value: "jenkins-headroom"
    effect: "NoSchedule"
  - key: "noexecute"
    operator: "Equal"
    value: "jenkins-headroom"
    effect: "NoExecute"
  nodeSelector:
    service: jenkins-headroom