Beehive Airflow¶
Archived (pre-2022)
Preserved for reference only -- likely outdated. View original | Last updated: September 2018
General Overview¶
As mentioned in the 'Beehive Infrastructure' part, the Beehive architecture setup includes three components (besides to the Bidding Agent) that run as Airflow jobs.
What is Airflow¶
Airflow is free open source Apache project was created to create and schedule workflows in programmatic way. Airflow defines DAG (Directed Acyclic Graphs) that directs tasks executed by a scheduler (as cron in specific time windows time). Workflow are treated as code which can be versioned in repository (Github...).
To read more about Airflow, check here → .
Airflow - Beehive¶
Airflow Dashboard can be accessed at : Admin
As highlighted in the Airflow console, Beehive projects run the following DAGs:
Airflow - ASP¶
- adscore_preparator: Triggers a staging workflow to build an ASP container in ECS (Fyber RTB Production Env). The ASP container creates a CDB file that will be pushed to an S3 bucket (beehive-staging-artifact) in Fyber Core Staging Env.
- adscore_preparator_prod: Triggers a production workflow to build an ASP container in ECS (Fyber RTB Production Env). The ASP container creates a CDB file that will be pushed to an S3 bucket (beehive-production-artifact) in Fyber Core production Env.
The ASP Workflow source code can be found here:
- Staging: dags (Github)
- Production: dags (Github)
Airflow - LICP¶
- line_item_capper_prepartor: Triggers a staging workflow to build an LICP container in ECS (Fyber RTB Production Env). The LICP container creates a CDB file that will be pushed to an S3 bucket (beehive-staging-artifact) in Fyber Core Staging Env.
- line_item_capper_preparator_prod: Triggers a production workflow to build an LICP container in ECS (Fyber RTB Production Env). The LICP container creates a CDB file that will be pushed to an S3 bucket (beehive-production-artifact) in Fyber Core production Env.
The LICP Workflow source code can be found here:
- Staging: dag (Github)
- Production: dag (Github)
Airflow - LIDP¶
- line_item_data_prepartor: Triggers a staging workflow to build an LIDP container in ECS (Fyber RTB Production Env). The LIDP container creates a CDB file that will be pushed to an S3 bucket (beehive-staging-artifact) in Fyber Core Staging Env.
- line_item_data_preparator_prod: Triggers a production workflow to build an LIDP container in ECS (Fyber RTB Production Env). The LIDP container creates a CDB file that will be pushed to an S3 bucket (beehive-production-artifact) in Fyber Core production Env.
The LIDP Workflow source code can be found here:
- Staging: dags (Github)
- Production: dags (Github)
Airflow - Infrastructure¶
Airflow - Monitoring and Troubleshooting¶
Airflow Dags and Task workflows can be monitored from Grafana via the following dashboard:
Failed jobs/tasks can be investigated by pointing the Airflow dashboard and check initially Logs for the correspondent failed one. This can be achieved by selecting the Logs section for the Job under question. E.g:
The Logs section would not expose all details regarding the reason of failed task of DAG. This can be escalated by verifying the the status of the ECS cluster/TD running in each of the following EC2 instances:
Note: The instances IP list might be changed/updated at the time or developing the present page
Once logged in, check the running docker containers (if still not existed). LICP, LICP and ASP containers are designed to execute a simple task and exist:
