← All work

DATA & ML · CLOUD PLATFORM LAB

Testing release recovery in a GKE lab

I used synthetic fulfillment data to test a forecasting workflow. Then I built a controlled GKE API lab to practise delivery, metrics, and recovery from a bad container image.

50Ksynthetic orders
69.59%model test WAPE · still high
GKEcontrolled API delivery lab
FULFILLAI / DELIVERY ARCHITECTURE02
{ }SourceGitHub
→
✓Build & scanActions · WIF
→
◇ImageExact digest
GIT-BACKED DESIRED STATE↓
GKE AutopilotArgo CD reconciles the Helm workload
API LAB
Health → metrics → alerts → recovery
PROJECT CONTEXT

FulfillAI

I built a pipeline on synthetic fulfillment data, then set up a separate GKE API lab. I released a deployment pointing to a nonexistent image, then verified recovery.

  • PostgreSQL / dbt
  • GCP / GKE
  • Terraform
  • Argo CD
Independent project / controlled lab

Synthetic fulfillment data

I generated a deterministic fulfillment dataset with 50,000 synthetic orders. PostgreSQL and dbt organize the relational data and transformations. Redpanda and PySpark explore streaming, while MLflow and FastAPI support experiment tracking and serving.

The data is synthetic by design. It lets me test contracts and workflows without presenting the results as real customer outcomes.

Lower WAPE, higher RMSE

On 63,501 synthetic test rows, the hard-gated two-stage hurdle forecast lowered WAPE from 88.24% to 69.59% versus a rolling-28 baseline. Decisions and thresholds were frozen before the final chronological test.

Frozen synthetic demand benchmark · lower is better
MetricRolling-28 baselineHurdle model
WAPE88.24%69.59%
RMSE0.8367230.933770

RMSE got worse. WAPE also remained high. I keep both results visible because improvement on one metric does not make a model broadly better.

What Actions and Argo CD each handle

Terraform provisions the GCP infrastructure. GitHub Actions validates the source and infrastructure, builds and scans the container, authenticates through OIDC / Workload Identity Federation, and promotes an exact image digest to Git-backed desired state.

Argo CD reconciles the Helm workload on GKE Autopilot. Actions owns build and promotion; Argo CD owns deployment. That distinction became useful when troubleshooting field ownership and rollout verification.

Recovering from a bad image

I introduced a nonexistent image and inspected ErrImagePull and ImagePullBackOff while healthy replicas continued serving. I restored a known-good release, checked recovery and restored Argo CD self-healing after the exercise.

Prometheus, Grafana and Alertmanager cover the observability path. A synthetic alert was routed end to end to Slack. The 99.9% SLO is a lab target used by alert rules, rather than a measured production uptime result.

What I verified in GKE

The recorded GKE checks cover API health, metrics, delivery and recovery. They do not establish that every model artifact or the full PostgreSQL, dbt and streaming stack ran in GKE.

HPA, PDB and RBAC exercises also include a separate local kind lab. NetworkPolicy objects and selectors were checked, but packet-level enforcement was not proved. The complete V2 implementation remains on platform-engineering-v2.

Source & evidence

Code, experiment results, and implementation notes.

NEXT CASE STUDYOTTO forecasting