Orchestrating Models, Not Just Containers: Kubernetes in the Age of AI

Most of my work lives at the application layer. It is where products are won: the flows, the APIs, the parts users can point at. Plenty of products sit directly on a single provider, one OpenAI or Anthropic model behind one key, and are perfectly well served by that. But once a product outgrows one model, or starts touching its own hardware, it comes to stand on a layer its users have never heard of: the one that decides, in the moment after they press enter, which model answers them, on which piece of hardware, and what happens if that hardware is saturated or gone. Nobody thanks that layer when it works. It gets exactly one requirement, and it is non-negotiable: it cannot fail the people who do not know they are using it.

And it is no longer only about the products we ship. It is how we build them. The coding tools on my machine make the same moves this article is about: Claude Code hands lightweight subtasks to a small fast model and keeps the hard reasoning on a big one, Cursor routes between its own models and frontier APIs depending on the action, and Anthropic's write-up of their multi-agent research system describes exactly this shape, with an Opus lead agent orchestrating cheaper Sonnet subagents. Model-per-job routing is quietly becoming how software gets made, not just how it gets served.

Meanwhile, teams everywhere are signing with frontier API providers (OpenAI, Anthropic, Google, xAI, MiniMax) while also hosting open-weights families like Llama, Qwen, DeepSeek or Mistral on whatever GPUs they managed to get, and someone has to orchestrate all those models across the hardware available. "How well do you know Kubernetes" is a 2019 question. The 2026 question is: given a fixed pool of GPUs, a set of open-weights models you want to host, and a set of frontier models you can only rent through an API, can you make all of it look like one boring endpoint? I built a working proof of concept that answers it: a model router, a set of GPU-aware deployments, and a demo that runs on a laptop. This post is the reasoning behind it.

The economics say utilization, loudly

Some context on why this specific skill suddenly matters. Blockworks' Friday Charts ran the numbers this week: hyperscalers and neoclouds will have spent around $2 trillion on capex by the end of the year, a single cabinet of GPUs in Nvidia's new Vera Rubin pods is estimated at $9 million, and the newsletter opens with Jensen Huang declaring that "the data center is the new unit of computing." The same issue notes that one gigawatt of data center capacity now produces nearly 500 trillion tokens, up from approximately zero in 2023.

You are probably not a hyperscaler, but the numbers stay painful at street level: a single high-end card can cost well over €40,000 before there is a server to put it in, and how to carve that one card up (whole, halved, sliced into sevenths) is a planning exercise of its own, which is why GPU vendors ship sizing guides and configuration matrices to walk buyers through the options.

To be clear, not every team needs to care, and it is not a binary choice between an API and your own cluster. It is a ladder, and each rung down hands you more control of what sits below the endpoint:

• All-API. An OpenAI or Anthropic model behind one key, or an Azure OpenAI deployment with its fixed model catalogue and token-per-minute quotas. Your levers are quotas, caching, and picking the right model tier per task.
• Rented inference. Hugging Face Inference Endpoints, RunPod and their peers, several of which I have used. You pick the model and the card, somebody else runs the machines, and the meter charges by the GPU-hour whether you are busy or idle.
• Cloud GPUs you operate. Raw GPU VMs billed by the hour on GCP, AWS or Azure (once the provider grants you quota, because the scarcity is real), or GPU node pools attached to a managed Kubernetes cluster like GKE, EKS or AKS, often at spot prices if your workload tolerates eviction. The meter looks like the rung above; the difference is that the orchestration below the endpoint is now yours.
• Metal you own. Cards bought and racked on-prem or in colocation: the €40,000 card from earlier, amortized instead of metered. It pays off with sustained utilization or hard data-residency constraints, and Kubernetes runs on bare metal too, so the same manifests follow you there.

From the second rung down the meter never stops, and the biggest lever a normal team fully controls becomes the idle fraction of the GPU-hour it already pays for. Orchestration is not a platform nicety; it is the difference between a fleet that earns its cost and a very expensive space heater. Kubernetes is not the only way to get there: Slurm still owns the HPC world, Ray ships its own scheduler, and every cloud will happily sell you a managed endpoint that hides the problem behind a margin. But if your product already runs on Kubernetes, the pragmatic move is to let the same platform bin-pack the models too: heterogeneous workloads onto heterogeneous machines, declaratively, next to the workloads it already runs.

Three forces, one design

Every multi-model setup I have seen is pulled apart by the same three forces:

• Models are heterogeneous. Some run on your GPUs (Qwen, Llama, and the rest of the open-weights world), some exist only behind an API (the frontier tiers, whoever you sign with). They differ in context window, tool support, cost, latency, and data-residency implications.
• Hardware is heterogeneous and finite. A real fleet is whatever you could get: a few A100s, a pool of L4s, H100s next quarter if the ordering gods smile. Each model has one or two placements that make economic sense, and the rest are waste.
• Product code wants none of this. It wants an OpenAI-compatible endpoint and a model name, and it should keep working when a provider is swapped, a contract is signed, or a card burns out.

The PoC splits responsibility along those lines. Kubernetes owns placement: which pod lands on which GPU, expressed in the language it already speaks (resource requests, node selectors, taints). A small router owns selection: which backend serves this request, with failover. A registry owns policy: what exists, what it can do, what it costs. One ConfigMap, hot-reloaded, so "we signed with a new provider" is a kubectl apply, not a product deploy.

Placement is where the money is

The part of the PoC I'd defend the hardest is its GPU playbook, because this is where Kubernetes knowledge and GPU knowledge have to meet:

Each row is a small argument. The 32B model does not fit one 80GB card with room to breathe (weights alone are ~64GB at bf16), so it is sharded across two, and the manifest pins the exact GPU product via nodeSelector, because "two GPUs somewhere in the cluster" and "two NVLinked A100s on one node" are wildly different machines. The 8B model goes the opposite way: quantize until it fits the card you can actually buy, then scale out. The memory freed by quantization is not just fit, it is KV cache, and KV cache is concurrency. Quantization often buys you throughput, not only smaller weights.

The embeddings row is my favorite, because it is where money quietly dies in most clusters. An embedding model needs about 2GB, and the same math applies to rerankers, guardrail classifiers, and Whisper-class transcription models, the small always-on servants every AI product accumulates. Give any of them a whole A100 and ~97% of a very expensive card idles forever. MIG, Multi-Instance GPU, carves the silicon into seven isolated slices; the Deployment requests nvidia.com/mig-1g.10gb: 1 like any other resource, and one card serves seven of them, each hardware-isolated from its neighbors. Time-slicing does something similar for dev clusters without any isolation at all: one greedy process can OOM the rest, which is exactly why that config in the PoC is labeled dev-only.

None of this is exotic. It is ordinary Kubernetes (resource requests, labels, probes, disruption budgets) pointed at an unfamiliar resource. That is the actual meaning of "leveraging Kubernetes in the age of AI": the platform did not change, the workload did.

The serving engine is a choice too

Everything in-cluster here is served by vLLM, and that is a tested preference, not a default. The property that sells it is counterintuitive: it gets better under pressure. Continuous batching means that the more concurrent requests you throw at an instance, the more tokens per GPU-hour it produces; low traffic is the pathological case, not the busy hour. I also tried Ollama along the way, and the comparison taught the same lesson as the placement table: tools encode assumptions about their workload. Ollama is excellent for what it is built for, which is one person and one machine, but under concurrent load it was clearly slower in my tests, and switching quickly between different models gave it real trouble, since every swap unloads one model and loads another. A vLLM instance never has that problem because it never changes clothes: one model per server for its whole lifetime, and the router above decides which server a request deserves.

Routing is a policy, not an if-statement

The router speaks /v1/chat/completions and resolves names in two deliberately different modes. An exact model name (qwen3-32b) follows an explicit, operator-written failover chain: the in-cluster vLLM first, a frontier API as overflow. No cleverness. An alias (chat-default, chat-long-context) is a set of requirements instead, minimum context window and tool support, plus a strategy: cheapest-healthy sorts every qualifying backend by blended cost; prefer-cluster keeps traffic on hardware you own unless nothing in-cluster qualifies, which doubles as the data-residency knob.

Aliases are the half that changes how a team works. Product code asks for an outcome, not a vendor, so "the 8B handles this traffic fine now" or "route long-context to whichever provider prices it better" become registry edits that ship in seconds. A per-backend circuit breaker skips unhealthy backends, every response carries an x-router-backend header so you always know who answered, and streaming fails over only before the first byte. Once tokens are flowing, re-running the generation behind the user's back would double the cost and splice two different answers together.

One thing the router refuses to do: queue. vLLM already maintains the honest queue, and hiding it behind a second one would destroy the best autoscaling signal in the system. Which brings me to the trap.

The GPU utilization trap

If you take one operational detail from this post, take this one: GPU utilization is a misleading autoscaling signal for LLM serving. Continuous batching keeps the card busy whether it is serving two requests comfortably or drowning under two hundred: both read ~100%. Scale on utilization and you will scale on noise.

The number that tracks what users feel is queue depth: vLLM's num_requests_waiting, requests admitted but not yet scheduled into the batch. The PoC scales the workhorse 8B tier on exactly that metric via KEDA: up fast, because queues compound; down slowly and one pod at a time, because every new replica pays minutes of model loading and flapping is expensive. And when the whole L4 pool is exhausted? That is not an autoscaling problem anymore. The failover chain overflows to the external APIs, the elastic tier you rent precisely because you do not own it.

The application layer does not get to pretend any of this away, either. A GPU bottleneck eventually surfaces in the product, and the honest responses live there: backpressure, timeouts, a degraded mode. A smaller model with an honest label beats a spinner that dies at sixty seconds.

What the PoC proves, and what it doesn't

The whole thing runs without a single GPU: a local overlay swaps every vLLM Deployment for a mock that speaks the same API and exports the same metrics, so routing, failover, and the scaling signals can be demonstrated on a laptop kind cluster. Kill the 32B deployment mid-demo and watch the same request come back answered by the "external" tier. The routing logic is pure and unit-tested; the manifests render through kustomize in CI.

It is still a PoC, and I'd rather say so than have you discover it: there is no authentication, no rate limiting, no per-tenant budgets, and the external keys are plain Secrets. That is the first production iteration, and none of it changes the architecture. The platform concepts transfer almost unchanged between distributions and clouds; what decides success is whether the workload's economics were understood before the YAML was written.

The application layer is still where I spend most of my days, and still where products are won. But underneath the AI products that grow past one model there is now a layer doing the hardest infrastructure job of the decade: taking a $2-trillion-capex-era constraint, a zoo of models, and a fixed pile of hardware, and making them behave like one reliable, swappable, boring endpoint, for people who will never know it exists. It has to work. It cannot fail them. The PoC is my way of showing it doesn't have to.