Datascience NLB migration.¶

Traffic flow diagram¶

Outside requests → GCP NLB Internal (L4) → Envoy rAVE sidecar Service (L7 GKE svc) → grpc rAVE container.

Application IaC¶

ArgoCD basically, there is a materialized config:
Argocd - Rave Dsp Live
and in DS repo helm chart templates:
appgrowthplatform (Gitlab)
It's split into two parts:

• static part loaded on boot by envoy (basically general stuff and connection to local rAVE)
• reloadable part that can be hot-swapped by Envoy - basically extra cluster definition and mirroring for Dark Traffic

Migration plan¶

DS Team decided to use Network Load Balancer instead of the Gateway API (L7).
They will use Envoy LB service as side-car container for rAVE application.

Migration plan:

1. Enable Subsetting for L4 Internal Load Balancers by this MR (apply could take ~40min)
   GKE module should be bumped to v3.5.3 !
2. Create a new Rave service with type: LoadBalancer and internal annotation. MR example

 annotations:
    networking.gke.io/load-balancer-type: "Internal"

1. Create a weighted DNS record to gradually switch traffic. 2nd Ip is from the new Load Balancer.
2. Validate that application is working by gcurl command from rAVE repo.

git clone git@gitlab.com:digitalturbine/appgrowthplatform/Shared/DS/rAVE.git
git submodule update --init --recursive

#### Validation script example

while true; do
ts=$(date '+%Y-%m-%d %H:%M:%S')
ips=$(dig +short rave-test.gke-core-datascience-dev-useast1.fyber.com A | paste -sd, -)
grpcurl -vv \
-import-path protocols/datascience/rave/proto/v1 \
-proto dsp.proto \
-d '{}' \
-authority rave-test.gke-core-datascience-dev-useast1.fyber.com \
-plaintext \
rave-test.gke-core-datascience-dev-useast1.fyber.com:80 \
dt.datascience.rave.v1.RaveService/Predict > /tmp/grpc.out 2>&1
rc=$?
echo "$ts resolved_ip=$ips status=$([ $rc -eq 0 ] && echo OK || echo ERR) rc=$rc"
[ $rc -ne 0 ] && tail -n 10 /tmp/grpc.out
sleep 1
done