← Dictionary / All-to-All Communication

All-to-All Communication

Appears in 1 paper

A communication pattern where every GPU sends data to every other GPU.

As used in Paper 19 — Ring Attention with Blockwise Transformers for Near-Infinite Context →

A communication pattern where every GPU sends data to every other GPU. O(P²) complexity. Ring Attention avoids this by using a ring topology (each GPU communicates with only 2 neighbours).

Appears in papers

Paper 19 — Ring Attention with Blockwise Transformers for Near-Infinite Context →