TG
All posts
2 min read

Slashing Commit-to-Production Lead Times by 80% on AKS

Helm templating, SHA-tagged images, OIDC deployment with no static credentials, and an HPA driven by queue depth instead of CPU.

At TradrLab, a commit to the AI-driven algorithmic trading platform took far too long to reach production on Azure Kubernetes Service. The stack was mixed — React, C#, and Python microservices — and the bottleneck was not compute. It was YAML: every service carried hand-maintained Kubernetes manifests applied with kubectl, and they drifted apart the moment anyone edited one.

Templating the manifests with Helm

I re-architected the deployment flow around Helm charts. Templating the manifests moved per-service detail into a single values.yaml, so changing an environment variable, a resource limit, or a replica count no longer meant editing Kubernetes objects by hand. Application developers stopped touching them entirely.

COMMIT TO PRODUCTION ON AKSSOURCEmain branchAZURE PIPELINES · CIAZURE PIPELINES · CDAKS PRODUCTION CLUSTERMETRICSGit commitReact · C# · PythonLint + unit testscode quality checksDocker buildaggressive layer cachingImage tagimmutable, Git SHAAzure Container Registryprivate registryOIDC auth to AKSno static credentialshelm upgrade--set image.tag=SHAIngress controllerHTTPS, external trafficMicroservice podsC# · PythonHPAcustom metrics, not CPUPrometheusqueue length · throughput1. push2. tests pass3. tag4. push image5. image available6. token7. rolling release8. route traffic9. export metricscustom metrics APIscale replicasnumbered happy pathscaling feedback
The autoscaler reads queue length and throughput from Prometheus rather than CPU, so bursty Arrow Flight loads scale before latency moves.

How a commit reaches production

  1. A developer pushes to main.
  2. Linting and unit tests run first, so a broken commit never reaches the build.
  3. The Docker build runs against an aggressive layer cache, and the image is tagged with the Git SHA — immutable, and traceable back to the exact commit.
  4. That tagged image is pushed to Azure Container Registry.
  5. The CD stage picks it up from the registry.
  6. It authenticates to the cluster over OIDC, so there are no static credentials stored in the pipeline.
  7. helm upgrade --install --set image.tag=<SHA> renders the chart and rolls the release out.
  8. An ingress controller fronts the cluster and routes external HTTPS traffic to the microservice pods.
  9. The pods export queue length and throughput to Prometheus, which the Horizontal Pod Autoscaler reads through the custom metrics API to set the replica count.

If Prometheus stops publishing, the HPA cannot compute a target and holds the replica count it already has. Pods keep serving; a burst simply queues.

Cache-busting on the lockfile, not the commit

The build cache only helps if it is invalidated for the right reason. Cache-busting is keyed to requirements.txt and package.json, so a dependency change rebuilds the layers below it and a code change does not. Most commits touch neither, and skip that work entirely.

Scaling on the signal that moves first

Market data arrives in bursts, and the platform handles large Apache Arrow Flight workloads. CPU is a lagging indicator of that load — by the time it climbs, requests are already waiting. Pointing the HPA at queue length and throughput instead means capacity arrives while the queue is still short, which is the difference between absorbing a burst and riding it out.

Together with the build caching, these changes cut commit-to-production lead time by 80%.

GitHub
LinkedIn