Kafka Monitoring¶
Imported from Confluence
Content may be outdated. Verify before following any procedures. View original | Last updated: April 2022
We have multiple Kafka clusters deployed with different approaches in AWS. We would like to unify them in the nearest future and use Strimzi Kafka operator as only one tool to deploy the Kafka cluster.
Below is the list of Kafka clusters with links for configuration and monitoring.
Kafka metric monitoring is done via Prometheus/Grafana, alerts with Prometheus alertmanager.
Fairbid¶
| Cluster name | Config | Monitoring | Description |
|---|---|---|---|
| Kafka SDK Events | fairbid_production_virginia (Bitbucket) | Grafana - Eks Fairbid Kafka | Kafka is used currently for SDK events. Will be replaced with Strimzi soon. |
| Kafka SDK Events Virginia | fairbid_production_virginia (Bitbucket) | Grafana - Kafka Strimzi Fairbid Virginia | Kafka for SDK events. Deployed with Strimzi. |
| Kafka Marketplace Ohio | bln_fairbid_production_ohio (Bitbucket) | Grafana - Kafka Strimzi Fairbid Ohio | Kafka for Marketplace events. Deployed with Strimzi in Ohio. |
| druid-metrics | fairbid_production_virginia (Bitbucket) | Grafana - Strimzi Kafka | Kafka used for druid mtrics. Deployed with Strimzi. |
Offerwall¶
| Cluster name | Config | Monitoring | Description |
|---|---|---|---|
| kafka legacy | app_kafka (Github) kafka_server (Github) | Grafana - Aws Kafka Core Grafana - Aws Kafka Core Topics | Kafka used for Core Engine. Deployed with terraform and legacy chef. |
| druid metrics | ofw_production_ireland (Bitbucket) | TBD | Kafka deployed for Offerwall druid metrics. Deployed with Strimzi. |