AWS has launched Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native system designed to improve how inference requests are distributed across GPU clusters.
Instead of relying on standard round-robin routing, the gateway evaluates real-time signals including KV cache utilization, queue depth, active requests, LoRA adapter availability, and prefix cache hits before selecting a model-serving pod.
AWS benchmarks showed P95 first-token latency reductions of up to 98% under some workloads and throughput improvements of up to 50%. The gateway supports OpenAI-compatible model servers, multi-model routing, existing Kubernetes tooling, and deployment as an EKS managed add-on without requiring application code changes.
