Autoscaling

Appears in 1 tutorial

Adjusting replica count to match traffic; for LLMs, scale on queue length / KV-cache utilization / TTFT-vs-SLO, not CPU.

As used in LLM Infrastructure →

Adjusting replica count to match traffic; for LLMs, scale on queue length / KV-cache utilization / TTFT-vs-SLO, not CPU. Complicated by slow model-load scale-up.

LLM Infrastructure →