Cross-Entropy Loss

Appears in 2 papers · 1 tutorial

The standard loss function for language modeling.

As used in Paper 12 — Language Models are Few-Shot Learners →

The standard loss function for language modeling. Measures how well the model's predicted probability distribution matches the true distribution (i.e., the correct next token). Lower loss = better model.

As used in Paper 13 — Scaling Laws for Neural Language Models →

The standard metric for language models. Measures how well the model's predicted probability distribution matches the true distribution. Lower loss = better model. Ranges from 0 (perfect) to infinity (terrible).

As used in Fine-Tuning & Model Customization →

The standard loss for language models; high when the model gave low probability to the correct token. (M03)

Paper 12 — Language Models are Few-Shot Learners → Paper 13 — Scaling Laws for Neural Language Models → Fine-Tuning & Model Customization →