0. Tech Stack¶

Containers¶

• • Kubernetes
  • Control Plane - Amazon EKS
  • Kubernetes Nodes - spot.io ocean
  • Metrics collection
  • Monitoring system - Prometheus
    • Short term metrics storage - Prometheus TSDB
    • Long Term Storage - Thanos with AWS S3
    • Visualisation - Grafana
    • Alerting - Alertmanager
  • Logs Collection
  • Logging Platform - Loki
  • Visualisation - Grafana
  • Collector - Promtail
  • Build and Deployment
  • CI/CD - Jenkins
  • Templating - helm
  • Helm structure and charts organisation - helmfile
  • Helm chart storage - git
  • Images - docker
  • Canary - flagger
  • Ingress
  • Nginx
  • AWS Load Balancer Controller
  • Operators
  • Spark operator
  • Strimzi Kafka Cluster Operator
  • Cert-manager
  • Tools
  • helm
  • helmfile
  • kubectl
  • lens

Infrastructure¶

• Deployment
• Terraform
  • TF Code organisation and state management
  • Terragrunt
  • Code storage
  • git
  • States
  • Storage
    • AWS S3
  • Locks
    • DynamoDB
• Cloud Providers
• AWS
• spot.io
• DNS Services
• DNS Made Easy (NS)
• INWX (registrator)
• CDN
• Akamai
• CloudFlare
• WAF
• ClaudFlare
• Certificates
• AWS ACM
• Digicert
• External Monitoring
• Pingdom

Automation¶

• CI/CD
• Jenkins - fully as a code
  • Jenkins Configuration as a Code plugin
  • Jenkins DSL plugin
  • Jenkins Kubernetes Cloud Plugin
  • Jenkins Pipelines

Monitoring¶

• Visualisation
• Grafana
• Metrics
• Prometheus
• CloudWatch
• Alerting
• Prometheus Alertmanager
• Grafana
• Cloudwatch
• Oncall
• OpsGenie
• External Monitoring
• Pingdom

Data Pipelines¶

• Scheduler and workflow manager
• Airflow in Kubernetes with workers on demand
  • Deployment
  • Helm chart - helm (Bitbucket)
  • DAGs - jenkins sync dags from git to EFS
• Data Processing Engine
• Spark
  • Deployment
  • Spark operator in Kubernetes
  • Processing Models
  • Spark Batch Processing
  • Spark Streaming
• Analytics
• Database
  • Apache Druid
• Reports and visualisation
  • Turnilo
  • Redash
  • Databricks

Databases¶

• MySQL
• Self-managed MySQL 5.6
• AWS RDS
• Postgresql
• AWS RDS
• Apache Druid

Message Brokers and Streaming Platforms¶

• Kafka
• Deployment
  • Kubernetes
  • EC2
• RabbitMQ
• Deployment
  • Kubernetes