← Dictionary / Token

Token

Appears in 2 papers

A unit of text, roughly a word or subword.

As used in Paper 12 — Language Models are Few-Shot Learners →

A unit of text, roughly a word or subword. GPT-3 uses a vocabulary of ~50,000 tokens. Longer sequences (more words) = more tokens = higher computational cost. The context window (~2,000 tokens) limits how much text you can input at once.

As used in Paper 13 — Scaling Laws for Neural Language Models →

A unit of text, roughly a word or subword. GPT-2 uses a 50K-token vocabulary; tokens are the building blocks of training data and model inputs.

Appears in papers

Paper 12 — Language Models are Few-Shot Learners → Paper 13 — Scaling Laws for Neural Language Models →