One inference pool. Three accelerator vendors. Twenty accelerators.

Accelerator fleets rarely remain uniform. Procurement arrives in phases, product generations overlap, supply changes, and different workloads reward different memory and compute characteristics. A platform that assumes permanent hardware uniformity eventually fights the data center it is supposed to manage.

Heterogeneity is already here

Single-vendor clusters are easier to reason about. Drivers, firmware, container images, kernels, collective libraries, and tuning guidance tend to move together. But operational simplicity is not the only constraint. Sovereign and on-premise environments also care about data residency, supply flexibility, long-term capacity economics, and using every accelerator already installed.

The platform challenge is to preserve the optimized software path for each accelerator while presenting one control plane for routing, policy, and observability. Flattening the differences would waste each platform’s strengths. Exposing every difference to applications would make the service impossible to consume. The useful abstraction lives between those extremes.

The cluster we built

Our test environment was one OpenShift AI cluster on the NxtGen Sovereign Cloud. Four nodes contributed twenty accelerators across three hardware families.

All nodes shared a 100 Gbps RoCE network. We pinned each vLLM replica to one accelerator with tensor parallelism set to one. That gave the routing layer twenty independent serving endpoints and made scheduling behavior visible instead of hiding it behind multi-accelerator replicas.

One service without pretending the hardware is identical

Each accelerator pool ran as a separate Helm release. Node selection, container images, and a small set of runtime settings remained vendor-specific. The replicas still carried common labels and registered into one llm-d InferencePool.

For the comparison, the same pods were also placed behind a Kubernetes ClusterIP service. That gave us a clean baseline: identical inference replicas and serving flags, with only the routing decision changing between Kubernetes round robin and llm-d.

We served two very different models:

• ibm-granite/granite-4.1-8b, an 8 billion parameter dense decoder model.
• sarvamai/sarvam-30b, a 30 billion parameter multilingual mixture-of-experts model.

Endpoint count is not capacity

Round robin treats every ready endpoint as an equal next choice. A heterogeneous pool makes the weakness obvious. Different accelerators reach saturation at different times, and a busy or lower-capacity replica can accumulate queue depth while another replica still has useful headroom.

llm-d’s endpoint picker considers saturation and prefix-cache locality. Requests sharing a long prompt can return to a replica where that prefix is already warm instead of repeatedly paying the prefill cost elsewhere. The router is not erasing hardware differences. It is making decisions with enough context to use them.

The workload favored intelligent routing

We used the prefill-heavy shared_prefix_synthetic workload from inference-perf: approximately 7.2 thousand input tokens followed by 1 thousand output tokens. A long shared system prompt makes it representative of RAG, chat, and citizen-service patterns where many requests reuse common context.

Important caveat: This is a favorable workload for prefix-cache-aware routing. Once a prefix becomes warm, llm-d can approach a very high cache-hit rate. Workloads with little prefix reuse should not be expected to reproduce the same gains.

What changed when routing became state-aware

Across the single-vendor pools and the combined pools, llm-d improved both output throughput and time to first token compared with the Kubernetes round-robin baseline. The benefit became larger as pool size and hardware diversity increased.

The limits taught us as much as the gains

We did not include Sarvam 30B results on the Intel pool. The model integration pinned vLLM 0.15.0, while the tested Intel runtime image used vLLM 0.16. A benchmark is more honest when an incompatible path is marked incomplete instead of being forced into the chart.

The all-vendor rate ladder stopped at 85 QPS because the single endpoint-picker replica became CPU-bound. Its work grows with both pod count and request rate. The accelerator pool itself had more room. Scaling the endpoint picker horizontally or assigning it more CPU should move that ceiling.

That bottleneck is a useful architectural result. Once the data plane becomes diverse and large enough, the control plane must scale with it. Intelligent scheduling is real work, and its own resource envelope belongs in capacity planning.

What this means for sovereign AI infrastructure

For me, the strongest outcome is optionality. A sovereign cloud should not need to discard useful capacity because the next procurement cycle selects a different accelerator. It should be able to keep data local, preserve vendor-optimized runtimes, and expose mixed capacity behind a consistent service contract.

This was not an accelerator ranking. Hardware generation, memory capacity, runtime maturity, and expected per-device performance all differed. The benchmark asked a more operational question: can one serving layer route effectively across the fleet that already exists? In this workload, the answer was yes.

Where I want this to go next

The next interesting step is cross-accelerator prefill and decode disaggregation. Prefill is compute-heavy. Decode is often shaped more by memory bandwidth. A future control plane could place each phase on the accelerator class that handles it best, even when those phases run on different vendor hardware.

That requires portable KV-cache transfer across accelerator ecosystems, stronger topology awareness, and a serving policy that understands cost as well as latency. It is difficult systems work. It is also exactly the kind of boundary where heterogeneous infrastructure stops being a compromise and becomes an advantage.

Further reading

• Original coauthored llm-d report · full methodology, charts, and results
• The Fifth Elephant · Pravein Govindan Kannan presents the conference perspective
• llm-d documentation · inference scheduling and control plane
• llm-d source code · CNCF Sandbox project