← Dictionary / Context window

Context window

Appears in 2 papers

The maximum number of tokens the model can see at once.

As used in Paper 10 — Improving Language Understanding by Generative Pre-Training →

The maximum number of tokens the model can see at once. GPT-1: 512 tokens. GPT-2: 1024. GPT-3: 2048. Modern models: up to 1 million+.

As used in Paper 12 — Language Models are Few-Shot Learners →

The maximum number of tokens the model can attend to at once. GPT-3's context window is ~2,000 tokens (roughly 1,500 words). Longer documents must be truncated or split into chunks.

Appears in papers

Paper 10 — Improving Language Understanding by Generative Pre-Training → Paper 12 — Language Models are Few-Shot Learners →