Skip to content

Direct Prod GKE Core

Imported from Confluence

Content may be outdated. Verify before following any procedures. View original | Last updated: August 2024

GKE Update Procedure for Direct

This document outlines the steps required to update Google Kubernetes Engine (GKE) for the Direct environment, specifically focusing on Kafka, RedisBloom, and Scylla.

1. Coordinate with the Direct Team

Before proceeding with any updates, contact a member of the Direct team to:

  • Agree on the plan for restarting Kafka and RedisBloom.
  • Determine the appropriate timing to minimize service disruption.

2. Restarting Redis

To restart Redis, follow these steps:

  1. Restart the Redis Master Node:

  2. Ensure the master node is up and running.

  3. Verify that it is fully loaded with all necessary data.

  4. Restart Redis Slave Nodes:

  5. Only restart the slave nodes after confirming the master node is operational and has completed data synchronization.

3. Restarting Kafka

  • Proceed to restart Kafka as agreed with the Direct team.

4. Coordinate with the DS Team (AVE)

  • Contact the Data Science (DS) team responsible for AVE (Artificial Vision Engine) and coordinate the timing for restarting Scylla.

5. Updating Scylla in GKE

To update Scylla in GKE:

  1. Create a New Node Pool:

  2. Set up a new node pool with the updated GKE version.

  3. Migrate Scylla Nodes:

  4. Gradually move each Scylla node to the new node pool, one at a time, ensuring minimal disruption to services.