Module 07 — Distributed & Multi-GPU Serving

Goal of this module: Learn what to do when one GPU isn’t enough — because the model is too big to fit, or you need more throughput than one card delivers. You’ll understand the four kinds of parallelism (tensor, pipeline, data, and expert), how GPUs talk to each other (interconnects), the costs each approach adds, and how to decide which to use. By the end you can reason about serving a 70B or 400B model across many GPUs.

This module attacks the “models are huge” problem from the other direction than quantization (Module 04): instead of shrinking the model, we spread it out.

7.1 Why one GPU runs out

Two independent reasons push you past a single GPU:

The model doesn’t fit. From Module 01: a 70B model is ~140 GB in FP16, but the biggest single GPUs today hold ~80–192 GB. Even quantized, the largest models (e.g. 400B+) exceed any single card. The weights plus KV cache simply don’t fit.
One GPU isn’t fast enough. Even if it fits, a single GPU’s memory bandwidth caps decode speed (Module 01) and its memory caps how many users you can batch. To serve more traffic or go faster, you add hardware.

These two reasons call for different parallelism strategies, which is why there are several. The first question to always ask: “Am I splitting because it doesn’t fit, or because I need more throughput?” The answer points you at the right technique.

7.2 The four kinds of parallelism (the core map)

TENSOR PARALLELISM (TP)   — split each layer's math ACROSS GPUs
   one model, every GPU holds a SLICE of every layer
   GPUs cooperate on EVERY token, talking constantly
   → use to FIT a big model + add bandwidth, within one fast-networked node

PIPELINE PARALLELISM (PP) — split the model BY LAYERS across GPUs
   GPU0 holds layers 1–20, GPU1 holds 21–40, ...
   a token flows through GPUs like an assembly line
   → use to FIT a very big model ACROSS MULTIPLE NODES

DATA PARALLELISM (DP)     — full COPY of the model on each GPU/replica
   each replica serves different requests independently
   → use to add THROUGHPUT when the model already fits on one GPU

EXPERT PARALLELISM (EP)   — for Mixture-of-Experts models
   distribute the model's "experts" across GPUs
   → use for large MoE models (Module 2.8)

Read that map twice. The whole module is elaborating it. The key split: TP and PP make a too-big model fit; DP adds throughput for a model that already fits; EP is a special case for MoE. They also combine — large deployments use TP and PP and DP together.

7.3 Tensor parallelism (TP): split each layer across GPUs

The idea: every layer of the model is mostly big matrix multiplications (Module 02). You can chop each matrix into pieces and put one piece on each GPU. Every GPU does its slice of every layer’s math, then the GPUs combine their partial results before moving on.

Analogy: four accountants splitting one giant spreadsheet column-wise. Each does their columns, then they add up the pieces to get the real total — every step, for every calculation.

What it gives you:

A model too big for one GPU now fits, because each GPU only stores a fraction of every layer’s weights and KV cache.
It also adds memory bandwidth: with the weights spread over 4 GPUs, each token’s weight-read is shared across 4 memory systems → faster decode (helps the memory-bound bottleneck from Module 01).

The cost — communication. Because the GPUs must combine partial results at every layer for every token, TP requires constant, heavy, low-latency communication between the GPUs. This is fine when the GPUs are in the same machine connected by a very fast link (NVLink), but becomes a bottleneck across slower links or separate machines. Therefore: keep tensor parallelism within a single node (one server with multiple GPUs wired by NVLink). This is the most important practical rule about TP.

In vLLM: --tensor-parallel-size N (Module 06). To serve a 70B model on a box with 4×80 GB GPUs, you’d set --tensor-parallel-size 4, and vLLM shards it for you.

7.4 Pipeline parallelism (PP): split by layers, like an assembly line

The idea: instead of slicing each layer, give whole layers to different GPUs. GPU 0 holds the first chunk of layers, GPU 1 the next chunk, and so on. A request flows through them in sequence: GPU 0 does its layers and hands the result to GPU 1, which does its layers and hands off to GPU 2, etc.

Analogy: a factory assembly line. Each station (GPU) does part of the work and passes the product along.

What it gives you:

Lets a model span multiple machines (nodes), because the hand-off between stages happens only at the boundaries between layer-chunks — far less communication than TP, and it tolerates slower links between nodes. So PP is how you go across nodes when a model is too big even for one full node of TP.

The cost — the “pipeline bubble.” In a naïve pipeline, while GPU 0 works on the first stage, GPUs 1–3 sit idle waiting for it; then GPU 0 idles while later stages work. These idle gaps are bubbles, and they waste capacity. The fix is to keep many requests/micro-batches in flight so every stage always has something to work on (like an assembly line that’s never starved). Good engines manage this, but PP generally needs healthy concurrency to be efficient.

In vLLM: --pipeline-parallel-size N, often combined with tensor parallelism — e.g. TP within each node, PP across nodes.

7.5 Combining TP and PP: the standard large-model layout

For the biggest models you use both, matched to the hardware hierarchy:

NODE 1 (8 GPUs, NVLink inside)      NODE 2 (8 GPUs, NVLink inside)
┌───────────────────────────┐      ┌───────────────────────────┐
│  TP across these 8 GPUs    │ ──►  │  TP across these 8 GPUs    │
│  (fast NVLink, holds       │  PP  │  (holds layers 41–80)      │
│   layers 1–40)             │ link │                            │
└───────────────────────────┘      └───────────────────────────┘
   tensor-parallel-size = 8            pipeline-parallel-size = 2

The principle: use the heavy-communication technique (TP) where the link is fastest (inside a node, over NVLink); use the lighter-communication technique (PP) where the link is slower (between nodes, over the network). Match the parallelism to the interconnect. That sentence is most of what you need to lay out a big deployment correctly.

7.6 Data parallelism (DP): replicate for throughput

The previous two split one model because it doesn’t fit. Data parallelism is different in purpose: the model already fits on one GPU (or one TP/PP group), and you want to handle more traffic. So you run multiple complete copies (replicas), put a load balancer in front, and spread incoming requests across them.

Analogy: the model fits in one checkout lane, but the line is too long, so you open more identical checkout lanes.

Each replica is independent — no per-token communication between replicas (unlike TP/PP). So DP scales throughput nearly linearly and is operationally simple.
This is the bread-and-butter way to scale out a fitting model under growing traffic, and it’s really an operations concern (load balancing, autoscaling) — picked up in Module 08.

Key distinction to keep crisp: TP/PP = “make a big model fit / run” (split one model). DP = “serve more requests” (clone a fitting model). Mixing these up is a classic beginner error. You can, of course, combine them: e.g. each replica is a TP group of 4 GPUs, and you run 5 such replicas (DP=5) behind a load balancer.

7.7 Expert parallelism (EP): for Mixture-of-Experts models

Recall MoE models (Module 2.8): the model has many “experts,” but each token only uses a few. MoE models can have enormous total parameters (so they need lots of memory to store) while being cheap to run per token.

Expert parallelism distributes the experts across GPUs — each GPU holds some experts. As tokens are routed to their chosen experts, work (and some communication to route tokens to the right GPU) is spread out. EP is the natural way to serve large MoE models and is often combined with TP/PP/DP. You’ll mostly encounter it as a configuration option when serving big MoE models; the concept to retain is “MoE → experts spread across GPUs.”

7.8 Interconnects: how GPUs talk (and why it dominates design)

The recurring theme above is communication cost, so you must know the hardware that carries it:

NVLink (and NVSwitch): NVIDIA’s very high-bandwidth, low-latency direct GPU-to-GPU links inside a server. Hundreds of GB/s to TB/s between GPUs. This is what makes tensor parallelism viable. When you see “8×GPU NVLink node,” that’s a box where all 8 GPUs are richly interconnected.
PCIe: the slower, general-purpose bus connecting GPUs to the system (and sometimes to each other) when NVLink isn’t present. Much lower bandwidth than NVLink — TP over PCIe-only links is often painfully slow.
Inter-node networking (InfiniBand / high-speed Ethernet, RDMA): the network between servers. Fast versions (InfiniBand, RoCE) are what make multi-node PP — and large-scale training — practical. Ordinary slow networking between nodes will bottleneck distributed serving.

The governing rule of distributed serving: parallelism that communicates a lot must run over the fastest link available. TP (chatty) → NVLink, inside a node. PP (less chatty) → can cross nodes over fast networking. DP (barely communicates) → can span anything, even data centers. Almost every layout decision falls out of matching communication intensity to interconnect speed.

7.9 The costs and downsides (so you don’t over-shard)

Distribution is not free. More GPUs working on one model adds:

Communication overhead: every split adds data movement; beyond a point, adding GPUs yields diminishing returns (and can even slow things if the interconnect is weak). Splitting a model that would fit on fewer GPUs just to use more GPUs often lowers efficiency.
Complexity & failure surface: more GPUs and nodes = more things to fail, harder debugging, trickier deployment (Module 08).
Cost: you’re paying for all those GPUs; utilization matters even more.

So the discipline is: use the least parallelism that meets your fit and performance needs. Quantize first if it lets a model fit on fewer GPUs (Module 04). Use TP within a node before reaching for PP across nodes. Add DP replicas for throughput rather than over-splitting a single model. “Can I avoid distributing further?” is a question an expert asks at every step.

7.10 A worked decision walk-through

“I need to serve a 70B model. What’s my layout?”

Does it fit on one GPU? 70B FP16 ≈ 140 GB > one 80 GB GPU. No.
Can quantization make it fit on one? INT4 ≈ 35 GB — yes, it could fit on one 80 GB GPU with room for KV cache. If quality is acceptable (Module 04), a single GPU with quantization may be the simplest answer — no distribution at all. Cheapest, simplest. Validate quality.
If you need FP16/FP8 quality or more bandwidth: use tensor parallelism within one node — e.g. --tensor-parallel-size 4 across 4×80 GB NVLink GPUs (≈140 GB of weights spread over 320 GB, plenty of KV room, 4× the bandwidth). This is the common production answer for 70B.
Need more traffic than that node serves? Add data-parallel replicas of that TP group behind a load balancer (Module 08). Scale replicas with traffic.
Model far bigger (e.g. 400B+) and won’t fit even one node? Add pipeline parallelism across nodes on top of TP within nodes.

Notice the order: quantize → TP in a node → DP replicas → PP across nodes, escalating only as forced. That ordering is the practical wisdom of this module distilled to one line.

Check your understanding

What are the two distinct reasons you’d go past a single GPU, and which parallelism types address each? (7.1, 7.2)
Why must tensor parallelism stay within a single node, while pipeline parallelism can cross nodes? (7.3, 7.4, 7.8)
What’s a pipeline “bubble,” and how do you reduce it? (7.4)
In one sentence each, contrast the purpose of TP/PP versus DP. (7.6)
State the governing rule that matches parallelism type to interconnect speed. (7.8)
For a 70B model, walk through the escalation order an expert would consider before committing many GPUs. (7.10)

Key terms introduced

node · tensor parallelism (TP) · pipeline parallelism (PP) · data parallelism (DP) · expert parallelism (EP) · replica · pipeline bubble · micro-batch · NVLink / NVSwitch · PCIe · InfiniBand / RoCE / RDMA · interconnect · tensor-parallel-size / pipeline-parallel-size

Next: Module 08 — Production Deployment & Operations, turning a running server into a reliable, observable, secure service.