Skip to content

Airflow UA

Archived (pre-2022)

Preserved for reference only -- likely outdated. View original | Last updated: April 2021

Airflow is a platform to programmatically author, schedule and monitor workflows. We used the official stable helm Chart from - airflow (Github)

Airflow is deployed in Ohio region:

Account URL DAGs folder
Fyber Core Ohio Airflow Ua - Admin airflow_common (Bitbucket)
  1. Add library/package, create new airflow image:

In order to add library/package to airflow create PRto DevOps team for approval, after approve you can build it with jenkins build job - Jenkins - airflow-build-job

Get image tag from the build job or appropriate ECR repo (767648288756.dkr.ecr.us-east-2.amazonaws.com/bln-airflow)

Updated values.yaml file in two places:

values.yaml (Bitbucket)

Redeploy airflow with Jenkins job - Jenkins - airflow-deploy-job

2. Dag update:

Put your dag in this folder airflow_common (Bitbucket)

Run Jenkins job - Jenkins - airflow-dags-deploy-job

UA Airflow depends on Redis, EFS - us-east-2 (Bitbucket)

Additional info:

Testing DAGs in EKS Airflow