Beehive CI/CD¶

General Overview¶

The Beehive project depends of the CI/CD processes to ensure a better software quality and in delivery manner. As mentioned in "Beehive Infrastructure" part, there are different developed components around the Beehive project that need:

• Version Control System
• Incorporation of code review
• Test and quickly build deliverable
• Track build results metrics
• Availability of the latest code release
• Ability to roll back in case of error during test
• Automation of build and deployment

That covers basically all Beehive components including:

• Bidding Agent
• Line Item Capper Preparator (LICP)
• Line Item Data Preparator (LIDP)
• Ad Scorer Preparator (ASP)

Build Pipelines - Overview¶

Beehive project requires separate environments with identical configuration. That exposes a production and staging environments. Both point to same code source repository.

CI build tool used for each environment is a Jenkins instance (running in AWS) and can be accessed as the following:

• Staging :      Jenkins
• Production:  Jenkins

Jenkins dashboard exposes several build pipelines for each Beehive component as follows:

• Bidding-Agent:  
  
• LICP: 
  
• LIDP:
  
• ASP:
  

Additionally, a secondary pipeline for the Bidding-Agent is created for rolling back to the previous and stable version:

• Bidding-Agent Roll Back:

Build Pipeline - Bidding Agent¶

The Bidding Agent build pipeline uses an additional plugin "multi-branch" . This will allow to handle branches automatically by creating different build when a new branch is pushed to github.

The complete code of the Jenkins pipeline for the Bidding agent can be accessed from the buy-side-tools repository inside the bidding agent source application code as follows:

• Staging:       Jenkinsfile (Github)

Please note that the merged branch will commit to the staging branch name. (master branch name in Jenkins Staging env is deprecated)

The Bidding Agent build/deploy pipeline in staging exposes mainly 10 stages (including 2 optional stages) as described in the Jenkinsfile:

• Checkout: Checkouts from the GitHub source code
• Test: Runs junit test
• Coverage Analysis: Publishes HTML reports based on generated builds based on Jenkins plugin HTML Publisher Plugin
• Prune Docker local files: Runs a local shell script [prune_local_docker_files.sh](https://github.com/SponsorPay/aws-infrastructure-code/blob/master/docker-images/jenkins/prune_local_docker_files.sh) to flush old built  local docker files in the builder worker that includes non used docker images and volumes.
• Build docker container: Builds and Wraps a docker container locally for Bidding Agent application code
• ECR Login: Logins to the ECR repository for a target region. This step requires a user input to choose which region the new deployment will take place as follows:
  
  Note: The user input stage is an additional plugin to the Jenkins build pipeline that can be found here→ for further reading.

• Publish to docker registry: Tags and pushes the new docker image to the ECR repository in the desired region. The new pushed docker image will be tagged with an additional Tag named "latest" referring as the latest image created and pushed in the repo:
  
• ECS Deploy: Deploys the new docker image in the running ECS cluster by triggering a task definition update. This stage requires a user input (approval or not). If yes, the stage will proceed to check if the deployment is triggered from the staging (for staging env) branch. If yes, the deployment could happen in 2 different modes:
• Blue Green Deployment: If 'BLUE_GREEN_DEPLOYMENT' env variable set to true, Jenkins worker will trigger a local script "bg_deploy.rb" which will perform a Blue/Green deployment based on the latest docker image update.
• ECS Deploy: that is the default deployment way which Jenkins worker triggers a local script "ecs_deploy.rb" that will run a set an updated task definition within the latest docker image pushed to ECR. The deployment will be handled by the ECS and the new BA version will be deployed and available automatically.

• System integration tests: Optional step that runs SIT agains the deployed version of the Bidding Agent container. By default SIT is disabled and can triggered if the "RUN_SYSTEM_INTEGRATION_TESTS" variable env is set to true.

• Clear old images in observed ECR registry:  Delete tagged images and keep the latest 5 images.

• Production:  Jenkinsfile_prod (Github)

Please note that the merged branch will commit to the master branch name in Jenkins Production env

The Bidding Agent build/deploy pipeline in production exposes mainly 10 stages (including 2 optional stages) as described in the Jenkinsfile_prod:

• Checkout: Checkouts from the GitHub source code
• Test: Runs junit test
• Coverage Analysis: Publishes HTML reports based on generated builds based on Jenkins plugin HTML Publisher Plugin
• Prune Docker local files: Runs a local shell script [prune_local_docker_files.sh](https://github.com/SponsorPay/aws-infrastructure-code/blob/master/docker-images/jenkins/prune_local_docker_files.sh) to flush old built  local docker files in the builder worker that includes non used docker images and volumes.
• Build docker container: Builds and Wraps a docker container locally for Bidding Agent application code
• ECR Login: Logins to the ECR repository for a target region. This step requires a user input to choose which region the new deployment will take place as follows:

Note: The user input stage is an additional plugin to the Jenkins build pipeline that can be found here→for further reading.

• Publish to docker registry: Tags and pushes the new docker image to the ECR repository in the desired region. The new pushed docker image will be tagged with an auto-do and additional Tag named "latest" referring as the latest image created and pushed in the repo:
  
• ECS Deploy: Deploys the new docker image in the running ECS cluster by triggering a task definition update. This stage requires a user input (approval or not). If yes, the stage will proceed to check if the deployment is triggered from the staging master (for production env) branch. If yes, the deployment could happen in 2 different modes:
• Blue Green Deployment: If 'BLUE_GREEN_DEPLOYMENT' env variable set to true, Jenkins worker will trigger a local script "bg_deploy.rb" which will perform a Blue/Green deployment based on the latest docker image update.
• ECS Deploy: that is the default deployment way which Jenkins worker triggers a local script "ecs_deploy.rb" that will run a set an updated task definition within the latest docker image pushed to ECR. The deployment will be handled by the ECS and the new BA version will be deployed and available automatically.
• System integration tests: Optional step that runs SIT agains the deployed version of the Bidding Agent container. By default SIT is disabled and can triggered if the "RUN_SYSTEM_INTEGRATION_TESTS" variable env is set to true.
• Clear old images in observed ECR registry:  Delete tagged images and keep the latest 5 images.

The Bidding Agent build pipeline for both staging and production environments can be summarised as illustrated in the following diagram:

Build Pipeline - LICP¶

The LICP build pipeline uses an additional plugin "multi-branch" (as for BA). This will allow to handle branches automatically by creating different build when a new branch is pushed to github.

The complete code of the Jenkins pipeline for the LICP can be accessed from the buy-side-tools repository inside the lineitem-capper-preparator source application code as follows:

• Staging:      Jenkinsfile (Github)

Please note that the merged branch will commit to the staging branch name. (master branch name in Jenkins Staging env is deprecated)

• Production:  Jenkinsfile_prod (Github)

As shown previously, the Jenkins worker will be building the LICP application code container and pushing it to the Fyber RTB ECR repository environment where it will be picked up and deployed via Airflow tasks.

The LICP build/deploy pipeline in both staging and production exposes mainly 7 stages as described in the Jenkins files (staging and production):

• Checkout: Checkouts from the GitHub source code
• Test: Runs junit test
• Coverage Analysis: Publishes HTML reports based on generated builds based on Jenkins plugin HTML Publisher Plugin
• Prune Docker local files: Runs a local shell script [prune_local_docker_files.sh](https://github.com/SponsorPay/aws-infrastructure-code/blob/master/docker-images/jenkins/prune_local_docker_files.sh) to flush old built  local docker files in the builder worker that includes non used docker images and volumes.
• Build docker container: Builds and Wraps a docker container locally for LICP application code
• ECR Login: Logins to the ECR repository for a target region. This step requires a user input to choose which region the new deployment will take place.
• Publish to docker registry: Tags and pushes the new docker image to the ECR repository in the desired region (Fyber RTB environment). The new pushed docker image will be tagged with an auto and additional Tag named "latest" referring as the latest image created and pushed in the ECR named line_item_capper_preparator.production for production and 'rtb-airflow-bees' for staging.

Build Pipeline - LIDP

The LIDP build pipeline uses an additional plugin "multi-branch" (as for BA). This will allow to handle branches automatically by creating different build when a new branch is pushed to github.

The complete code of the Jenkins pipeline for the LIDP can be accessed from the buy-side-tools repository inside the lineitem-data-preparator source application code as follows:

• Staging:      Jenkinsfile (Github)

Please note that the merged branch will commit to the staging branch name. (master branch name in Jenkins Staging env is deprecated)

• Production:  Jenkinsfile_prod (Github)

As shown previously, the Jenkins worker will be building the LIDP application code container and pushing it to the Fyber RTB ECR repository environment where it will be picked up and deployed via Airflow tasks.

The LIDP build/deploy pipeline in both staging and production exposes mainly 7 stages as described in the Jenkins files (staging and production):

• Checkout: Checkouts from the GitHub source code
• Test: Runs junit test
• Coverage Analysis: Publishes HTML reports based on generated builds based on Jenkins plugin HTML Publisher Plugin
• Prune Docker local files: Runs a local shell script [prune_local_docker_files.sh](https://github.com/SponsorPay/aws-infrastructure-code/blob/master/docker-images/jenkins/prune_local_docker_files.sh) to flush old built  local docker files in the builder worker that includes non used docker images and volumes.
• Build docker container: Builds and Wraps a docker container locally for LIDP application code
• ECR Login: Logins to the ECR repository for a target region. This step requires a user input to choose which region the new deployment will take place.
• Publish to docker registry: Tags and pushes the new docker image to the ECR repository in the desired region (Fyber RTB environment). The new pushed docker image will be tagged with an auto and additional Tag named "latest" referring as the latest image created and pushed in the ECR named 'line_item_data_preparator.production' for production and line_item_data_preparator for staging.

Build Pipeline - ASP¶

The ASP build pipeline uses an additional plugin "multi-branch" (as for BA). This will allow to handle branches automatically by creating different build when a new branch is pushed to github.

The complete code of the Jenkins pipeline for the ASP can be accessed from the buy-side-tools repository inside the adscorer-preparator source application code as follows:

• Staging:      Jenkinsfile (Github)

Please note that the merged branch will commit to the staging branch name. (master branch name in Jenkins Staging env is deprecated)

• Production:  Jenkinsfile_prod (Github)

As shown previously, the Jenkins worker will be building the ASP application code container and pushing it to the Fyber RTB ECR repository environment where it will be picked up and deployed via Airflow tasks.

The ASP build/deploy pipeline in both staging and production exposes mainly 7 stages as described in the Jenkins files (staging and production):

• Checkout: Checkouts from the GitHub source code
• Test: Runs junit test
• Coverage Analysis: Publishes HTML reports based on generated builds based on Jenkins plugin HTML Publisher Plugin
• Prune Docker local files: Runs a local shell script [prune_local_docker_files.sh](https://github.com/SponsorPay/aws-infrastructure-code/blob/master/docker-images/jenkins/prune_local_docker_files.sh) to flush old built  local docker files in the builder worker that includes non used docker images and volumes.
• Build docker container: Builds and Wraps a docker container locally for ASP application code
• ECR Login: Logins to the ECR repository for a target region. This step requires a user input to choose which region the new deployment will take place.
• Publish to docker registry: Tags and pushes the new docker image to the ECR repository in the desired region (Fyber RTB environment). The new pushed docker image will be tagged with an auto and additional Tag named "latest" referring as the latest image created and pushed in the ECR named 'adscore-preparator.production' for production and adscore-preparator for staging.

The LIDP, LICP and ASP have similar build pipelines for both staging and production environments can be summarised as illustrated in the following diagram:

TODO:

• LICP/LIDP/ASP built in staging and production or pushed to ECR Fyber RTB only Production Account (separate the workflow to use RTB Staging Account for staging images)