Interconnect

Any GPU-to-GPU or node-to-node link; its speed dictates which parallelism you can use where (the governing rule: chatty parallelism → fastest link).