Module 08 — Production Deployment & Operations

Goal of this module: Turn a working inference server into a real service that survives production: packaged in containers, deployed and scaled (often on Kubernetes), observed with proper metrics and logging, made reliable and safe to roll out, and secured. This is the difference between “I ran vLLM on a GPU once” and “we serve millions of requests a day without paging anyone at 3 a.m.”

This module is more breadth than depth — it surveys the operational surface so you know what exists and why. Each topic is a field of its own; the aim is fluency and good judgment, not mastery of every tool.

8.1 From “it runs” to “it’s a service”

A running vLLM process (Module 06) is not yet a production service. Production adds requirements that have nothing to do with the model:

It must start reliably anywhere, not just on your machine → containers.
It must scale up and down with traffic and recover from crashes → orchestration / autoscaling.
You must see what it’s doing → observability (metrics, logs, traces).
It must not fall over under load or bad input → reliability.
It must be safe to update → deployment strategies.
It must be secured → auth, network, input safety.
It must be affordable → cost controls (Module 09).

These are the “boring” parts that actually determine whether a deployment succeeds. Let’s walk them.

8.2 Containers: package once, run anywhere

The single biggest source of “works on my machine” pain in GPU serving is the environment: driver, CUDA, Python, library versions (you met this in Module 06). The fix is containers (Docker).

A container bundles your server, its exact dependencies, and config into one portable image that runs identically everywhere. For GPU work you use a GPU-enabled base image (NVIDIA provides CUDA base images; vLLM publishes official images too) and the NVIDIA Container Toolkit, which lets the container see the host’s GPUs.

A typical containerized vLLM service:

# Conceptual sketch — use the official vLLM image in practice.
FROM vllm/vllm-openai:latest          # vLLM + CUDA + deps, pre-built
# model can be baked in or (better) mounted/downloaded at runtime
ENV MODEL=meta-llama/Llama-3.1-8B-Instruct
EXPOSE 8000
ENTRYPOINT ["python", "-m", "vllm.entrypoints.openai.api_server"]
CMD ["--model", "meta-llama/Llama-3.1-8B-Instruct", "--gpu-memory-utilization", "0.9"]

Run it with GPU access:

docker run --gpus all -p 8000:8000 your-vllm-image

Key operational point — model weights are big. A 70B model is tens of GB. Baking weights into the image makes it huge and slow to deploy. Better: keep the image lean and load weights at startup from a mounted volume or object storage (S3/GCS), or use a shared fast filesystem. Model loading time becomes a real factor in how fast you can scale (Section 8.4).

8.3 Orchestration with Kubernetes (the standard)

One container is easy. Hundreds of containers across dozens of GPU machines, self-healing, load-balanced, and autoscaling — that needs an orchestrator. The industry standard is Kubernetes (K8s).

You don’t need to master Kubernetes to understand the serving-relevant pieces:

Pod: the smallest unit — your container(s) running together. A vLLM pod requests GPU resources (e.g. “1 GPU” or “4 GPUs” for a TP group).
Deployment / ReplicaSet: keeps N copies (replicas) of your pod running — this is data parallelism (Module 7.6) operationalized. If one dies, K8s restarts it.
Service / Ingress + Load Balancer: a stable address in front of all replicas that spreads requests across them.
GPU scheduling: K8s (with the NVIDIA device plugin) places GPU pods onto machines that have free GPUs, respecting how many each needs.
Health probes: liveness (is it alive? restart if not) and readiness (is it ready to serve? don’t send traffic until model is loaded) checks. Readiness probes matter a lot for LLMs because model loading takes a while — you don’t want traffic routed to a pod still loading 40 GB of weights.

There are also LLM-specialized layers built on K8s (e.g. KServe, Ray Serve, NVIDIA’s serving stacks, vLLM’s production-stack/router projects) that handle model loading, autoscaling, and routing with LLM-awareness. Know they exist; reach for them when raw K8s gets tedious.

8.4 Autoscaling: matching capacity to traffic

Traffic is bursty (Module 00). Running peak capacity 24/7 wastes money; running too little drops requests. Autoscaling adjusts the number of replicas to match load.

The subtlety for LLMs: what metric do you scale on?

CPU utilization (the classic web-app signal) is useless for GPU serving.
GPU “utilization” is misleading (Module 3.6).
Better signals: queue length / number of waiting requests, KV-cache utilization, TTFT trending toward your SLO, or requests-per-second per replica vs a known safe capacity. Scale up when these say you’re saturating; scale down when they say you’re idle.

The hard parts unique to LLM autoscaling:

Slow scale-up. A new replica must acquire a GPU and load tens of GB of weights before it serves — often minutes. So you must scale up early (predictively, or on leading indicators), keep warm pools, and not expect instant relief. This is why model-load time (Section 8.2) is an operational metric you actually track.
GPU scarcity/cost. GPUs may not be instantly available to scale onto, and they’re expensive — so autoscaling policy is also a cost decision (Module 09).

8.5 Observability: you can’t operate what you can’t see

From Module 03 you know which metrics matter (TTFT, TPOT, throughput, goodput, queue length, KV-cache utilization). Production is about collecting and watching them.

Metrics: vLLM exposes a /metrics endpoint in Prometheus format (Module 6.8). Prometheus scrapes and stores these; Grafana dashboards visualize them. Build dashboards for the Module 03 metric set, broken down by percentile (p50/p95/p99).
Logging: structured logs of requests (without leaking sensitive prompt content — see security), errors, and restarts. Aggregate them centrally so you can search across all replicas.
Tracing: for complex pipelines (e.g. RAG: retrieve → prompt → generate), distributed tracing shows where time goes across components.
Alerting: page when SLOs are at risk — e.g. “p99 TTFT > 1 s for 5 minutes,” “KV-cache utilization > 95%,” “error rate > 1%,” “replica crash-looping.” Alert on symptoms users feel (latency, errors), not just machine stats.

The operational loop: dashboards tell you what’s happening, alerts tell you when to act, and the metric definitions from Module 03 tell you what the numbers mean. Together they let you run the system instead of being surprised by it.

8.6 Reliability and graceful behavior under stress

Production traffic will exceed capacity sometimes. A reliable service degrades gracefully instead of collapsing:

Admission control / queue limits: cap the queue. When full, reject fast with a clear error (e.g. HTTP 429 “try again”) rather than accepting work you can’t serve, which would blow everyone’s latency. A fast, honest “no” beats a slow, silent timeout.
Timeouts and cancellation: if a client disconnects, stop generating for it — don’t waste GPU on output no one will read. (vLLM supports request cancellation; make sure your stack honors it.)
Max output length caps: bound max_tokens so a single runaway request can’t monopolize a slot forever (recall the static-batching pain in Module 5.2, and KV growth in Module 02).
Retries with backoff on the client side, but be careful: naive retries during an overload amplify the overload (“retry storm”). Use backoff and circuit breakers.
Health checks & auto-restart (Section 8.3) so a wedged process recovers itself.
Resource isolation: one tenant or one giant request shouldn’t starve others (rate limits per user/key).

The mindset: assume overload and bad input will happen, and design so the system bends instead of breaking.

8.7 Safe deployments: shipping updates without outages

You’ll update models and configs often. Doing it without downtime or regressions:

Rolling updates: replace replicas a few at a time so the service stays up throughout (K8s does this natively).
Blue-green: stand up the new version (green) alongside the old (blue), switch traffic over once green is verified, keep blue ready to roll back instantly.
Canary: send a small percentage of traffic to the new version, watch the metrics (latency and output quality), then ramp up if healthy. Essential for LLMs because a new model or quantization can change output quality, not just speed — and quality regressions are invisible to infra metrics. Evaluate quality on real traffic during canary (tie back to Module 04’s “test on your own task”).
Versioning & rollback: keep previous model versions and configs so you can revert in seconds. Pin versions; don’t deploy “latest” blindly.

8.8 Security and safety

Serving an LLM exposes several surfaces:

Authentication & authorization: require API keys/tokens; don’t run an open endpoint on the internet. vLLM’s --api-key is a start; real deployments use a gateway/auth layer. Apply per-key rate limits and quotas (also a cost control).
Network security: keep inference servers on private networks behind a gateway; don’t expose the raw vLLM port publicly. Use TLS for traffic.
Input/output safety: LLMs are subject to prompt injection (malicious instructions hidden in input, especially dangerous when the model can call tools or read untrusted documents in RAG) and can emit harmful or sensitive content. Mitigations include input/output filtering, guardrail/moderation models, constraining tool access, and never blindly trusting model output that drives actions.
Data privacy: prompts may contain sensitive user data. Be careful what you log (don’t dump raw prompts into logs that aren’t access-controlled), how long you retain it, and where it flows. This intersects with compliance (GDPR/HIPAA-type concerns) depending on your domain.
Multi-tenant isolation: if many customers share infrastructure, ensure one can’t see another’s data (e.g. be aware that prefix caching shares KV state — make sure shared cache can’t leak content across tenants in a way that matters for your threat model).

You don’t have to be a security expert, but you must know these exist and bring in the right people. Shipping an unauthenticated, unfiltered LLM endpoint is a real and common mistake.

8.9 Putting it together: a reference production shape

A common, sane production architecture combining this module:

                       Internet
                          │
                   ┌──────────────┐
                   │  API Gateway │  auth, rate limits, TLS, routing
                   └──────────────┘
                          │
                   ┌──────────────┐
                   │ Load Balancer│  spreads across replicas (DP)
                   └──────────────┘
              ┌───────────┼───────────┐
              ▼           ▼           ▼
        ┌─────────┐ ┌─────────┐ ┌─────────┐
        │ vLLM pod│ │ vLLM pod│ │ vLLM pod│   each = 1 GPU or a TP group
        │ (GPU)   │ │ (GPU)   │ │ (GPU)   │   autoscaled on queue/KV signals
        └─────────┘ └─────────┘ └─────────┘
              │           │           │
              └────────── /metrics ───┘──► Prometheus ──► Grafana + Alerts
                                            (TTFT, TPOT, goodput, queue, KV%)
        Weights loaded from object storage; readiness probes gate traffic;
        rolling/canary deploys; logs shipped to central store.

Every box here is a concept you now understand: replicas are data parallelism (Module 07), each pod runs vLLM with the flags from Module 06, the metrics are from Module 03, and the whole thing exists to meet an SLO at the lowest cost — which is Module 09.

Check your understanding

Why are containers the standard answer to “works on my machine” for GPU serving, and why is baking weights into the image a bad idea? (8.2)
In Kubernetes terms, what makes a readiness probe especially important for LLM serving? (8.3)
Why are CPU and even “GPU utilization” poor autoscaling signals for LLMs, and what better signals exist? (8.4)
Name two ways a service can degrade gracefully under overload instead of collapsing. (8.6)
Why is canary deployment especially important for LLMs compared to ordinary web services? (8.7)
Give two distinct security concerns specific to serving LLMs. (8.8)

Key terms introduced

container / Docker · NVIDIA Container Toolkit · Kubernetes / pod / deployment / replica · liveness vs readiness probe · KServe / Ray Serve · autoscaling (and LLM-aware signals) · warm pool · Prometheus / Grafana · observability (metrics/logs/traces) · alerting · admission control / rate limiting · retry storm · rolling / blue-green / canary deployment · prompt injection · guardrails / moderation · multi-tenant isolation

Next: Module 09 — Benchmarking, Cost & Capacity Planning, where you learn to measure honestly and turn performance into dollars and hardware decisions.