Strimzi¶

Archived (pre-2022)

Preserved for reference only -- likely outdated. View original | Last updated: September 2021

Overview¶

In order to unify out Kafka cluster deployment in Kubernetes Strimzi Kafka operator was chosen.

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components.

Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. It has following features:

Secured by default (Built-in security, TLS, SCRAM-SHA, and OAuth authentication, Automated Certificate Management)
Simple yet Configurable (NodePort, Load balancer and Ingress options, Rack awareness for HA, Use dedicated nodes for Kafka)
Kubernetes-Native Experience (Use kubectl to manage, Kafka Operator-based, Manage Kafka using GitOps)

The Kube-native management of Kafka is not limited to the broker. You can also manage Kafka topics, users, Kafka MirrorMaker and Kafka Connect using Custom Resources. This means you can use your familiar Kubernetes processes and tooling to manage complete Kafka applications.

Strimzi provides container images and Operators for running Kafka on Kubernetes. Strimzi Operators are fundamental to the running of Strimzi. The Operators provided with Strimzi are purpose-built with specialist operational knowledge to effectively manage Kafka.

Operators simplify the process of:

Deploying and running Kafka clusters
Deploying and running Kafka components
Configuring access to Kafka
Securing access to Kafka
Upgrading Kafka
Managing brokers
Creating and managing topics
Creating and managing user

General overview of Strimzi architecture:

Screenshot 2021-09-06 at 13.12.35.png

Deployment¶

Strimzi Kafka operator¶

Chart - v0.25.0 (Bitbucket)

Config - fairbid_production_virginia (Bitbucket)

Kafka Cluster¶

Chart - v1.0.1 (Bitbucket)

Config - fairbid_production_virginia (Bitbucket)

Topic as a code¶

Topics are stored as a code in the values.yaml file, to add a new topic or edit existing ones please modify next values.yaml - values.yaml (Bitbucket)

User as a code¶

TBD¶

Example¶

Fairbid SDK Events Kafka and Zookeeper are deployed with Strimzi operator:

LTMY-1509-DEBE:kafka-strimzi mguk$ kubectl get all -n kafka-sdk-events
NAME                                          READY   STATUS    RESTARTS   AGE
pod/fairbid-entity-operator-bc4ffd859-hsvvr   3/3     Running   0          6m57s
pod/fairbid-kafka-0                           1/1     Running   0          10m
pod/fairbid-kafka-1                           1/1     Running   0          10m
pod/fairbid-kafka-2                           1/1     Running   0          10m
pod/fairbid-kafka-3                           1/1     Running   0          10m
pod/fairbid-kafka-4                           1/1     Running   0          10m
pod/fairbid-kafka-exporter-6f49bc64d8-qqr6v   1/1     Running   0          6m15s
pod/fairbid-zookeeper-0                       1/1     Running   0          15m
pod/fairbid-zookeeper-1                       1/1     Running   0          15m
pod/fairbid-zookeeper-2                       1/1     Running   0          15m

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/fairbid-kafka-bootstrap    ClusterIP   10.100.60.4      <none>        9091/TCP,9092/TCP            10m
service/fairbid-kafka-brokers      ClusterIP   None             <none>        9090/TCP,9091/TCP,9092/TCP   10m
service/fairbid-zookeeper-client   ClusterIP   10.100.208.161   <none>        2181/TCP                     15m
service/fairbid-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   15m

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fairbid-entity-operator   1/1     1            1           6m58s
deployment.apps/fairbid-kafka-exporter    1/1     1            1           6m16s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/fairbid-entity-operator-bc4ffd859   1         1         1       6m58s
replicaset.apps/fairbid-kafka-exporter-6f49bc64d8   1         1         1       6m16s

NAME                                 READY   AGE
statefulset.apps/fairbid-kafka       5/5     10m
statefulset.apps/fairbid-zookeeper   3/3     15m

Kafka and zookeeper brokers are deployed on separate nodes in one VNG and same availability zone, according to the rule:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - us-east-1c
            - key: spotinst.io/node-lifecycle
              operator: In
              values:
                - od                          
            - key: fairbidsdk_kafka_strimzi_prd
              operator: In
              values:
                - fairbidsdk_kafka_strimzi_prd                          
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - bln-fairbid-strimzi
        topologyKey: "kubernetes.io/hostname"

Grafana dashboards are stored as a code in kafka repo - files (Bitbucket)

Dashboards are deployed and available in next Grafana folder - Grafana - Strimzi Zookeeper

Screenshot 2021-09-06 at 15.27.57.png