Papers / Language Models are Few-Shot Learners / Further Reading
Paper 12

Further Reading — Language Models are Few-Shot Learners

Language Models are Few-Shot Learners · 2020 · Tom Brown et al.

Further Reading: Paper 12 (GPT-3)

Deepen your understanding of GPT-3 and its context with these resources.

The Original Paper

Language Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, et al. (OpenAI)
Published: June 2020, NeurIPS 2020
URL: https://arxiv.org/abs/2005.14165

The full paper with all experiments, benchmarks, and detailed results. Dense but worth reading for the complete story. Focus on:

  • Section 3: “Tasks and Datasets” (shows all tasks tested)
  • Section 4: “Results” (performance across domains)
  • Section 5: “Limitations” (honesty about failure modes)

Essential Follow-Up Papers

1. Scaling Laws for Neural Language Models

Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, et al. (OpenAI)
Published: January 2020, arXiv
URL: https://arxiv.org/abs/2001.08361

Why does GPT-3’s scale matter? This paper studied power-law scaling relationships between model size, data size, compute, and performance. It predicted that GPT-3 would be the level of capability it achieved. This is Paper 13 in our series.

2. Language Models are Unsupervised Multitask Learners (GPT-2)

Authors: Alec Radford, Jeffrey Wu, Rewon Child, et al. (OpenAI)
Published: February 2019, preprint
URL: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

GPT-2, the predecessor to GPT-3. Much smaller (1.5B parameters) but showed that language models could do zero-shot multitask learning. GPT-3 scaled this idea up. Understanding GPT-2 helps understand GPT-3’s design.

3. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Authors: Jason Wei, Xuezhi Wang, Dale Schlarman, et al. (Google Brain)
Published: January 2023, arXiv
URL: https://arxiv.org/abs/2201.11903

GPT-3 struggles with multi-step reasoning. This paper showed that by asking the model to think step-by-step (“Let me work through this…”), performance on math and logic improves dramatically. A practical follow-up addressing one of GPT-3’s main limitations. Likely Paper 14 in our series.

Deeper Understanding

Blog Posts and Tutorials

“The Illustrated Transformer” — Jay Alammar
URL: http://jalammar.github.io/illustrated-transformer/
A visual, intuitive explanation of how Transformers work. Excellent for understanding the attention mechanism that makes GPT-3 possible.

“A Primer in BERTology: What We Know About How BERT Works” — Anna Rogers, Olga Kovaleva, Anna Rumshisky
URL: https://aclanthology.org/2020.emnlp-main.16/
While focused on BERT, this paper explains Transformer internals in detail. Helps understand why attention enables in-context learning.

“Prompt Engineering Guide” — DAIR.AI
URL: https://github.com/dair-ai/Prompt-Engineering-Guide
Comprehensive, community-maintained guide to prompt engineering techniques. Practical strategies for using GPT-3 and similar models.

“In-Context Learning and Induction Heads” — Catherine Olah’s Blog
URL: https://colah.github.io/ (and related mechanistic interpretability work)
Mechanistic studies of how Transformers implement in-context learning. Dense but fascinating for understanding the “why” behind GPT-3’s abilities.

Benchmarks and Datasets Used

GPT-3 was tested on 42 tasks. Key benchmarks include:

BenchmarkTaskReference
SUPERGLUEText understanding (classification, QA, similarity)https://super.gluebenchmark.com/
LAMBADAWord prediction in contexthttps://zenodo.org/record/2630551
DROPDiscrete reasoning over paragraphshttps://allennlp.org/drop
MATHMathematical problem solvinghttps://openai.com/blog/gpt-3/index.html
HumanEvalCode generation from docstringshttps://github.com/openai/human-eval
WinogradPronoun resolution (difficult)https://winograd.cs.washington.edu/

These benchmarks help you understand where GPT-3 excels and where it struggles.

Code and Models

Interactive Access

OpenAI Playground:
URL: https://platform.openai.com/playground
Try GPT-3 (and newer models like GPT-4) in a browser without code. Allows experimentation with temperature, max tokens, and prompts.

OpenAI API Documentation:
URL: https://platform.openai.com/docs/models
Full API reference for using GPT-3 programmatically. Pricing, rate limits, and best practices.

Open-Source Alternatives

If you want to run your own model (without API fees):

LLaMA and LLaMA-2 (Meta)
URL: https://ai.meta.com/blog/large-language-model-llama-meta-ai/
Open-source models (7B–70B parameters). Can be fine-tuned. Code available.

BLOOM (BigScience)
URL: https://huggingface.co/bigscience/bloom
176B parameters, multilingual, open-source. Trained by a collaborative research project.

Mistral (Mistral AI)
URL: https://mistral.ai/
Smaller but fast alternatives. 7B–12B parameters, permissive licensing.

Code Repos:

What Came Next

Direct Successors

InstructGPT (Ouyang et al., 2022)
GPT-3 fine-tuned with human feedback (RLHF) to follow instructions better. Intermediate step to ChatGPT.

ChatGPT (OpenAI, November 2022)
Fine-tuned InstructGPT for dialogue. The public version that brought LLMs to mainstream attention. Same core architecture as GPT-3, but much better at conversation.

GPT-4 (OpenAI, March 2023)
Multimodal (text + images), improved reasoning, longer context window. Size unknown but likely much larger than 175B parameters.

Constitutional AI (Bai et al., 2022)
An alternative to RLHF fine-tuning. Fine-tune with explicit principles (“Help the user, be honest, avoid harmful content”) instead of human examples. Useful for scaling alignment.

Self-Consistency Decoding (Wang et al., 2023)
Instead of asking for one answer, ask multiple times and take the majority vote. Improves reasoning accuracy.

Retrieval-Augmented Generation (RAG)
Combine language models with external knowledge (Wikipedia, documents). Solves the hallucination problem by grounding generation in facts.

Key Insights to Carry Forward

  1. Scale matters more than architecture: GPT-2 and GPT-3 use the same architecture. Scale unlocks new capabilities.

  2. Pre-training on diverse data is powerful: 300B tokens from the web gives implicit knowledge that enables few-shot learning.

  3. In-context learning is real: The model learns from examples in the prompt, not just from pre-training knowledge. Prompt format matters.

  4. Limitations are real: Hallucination, prompt sensitivity, weak reasoning are fundamental challenges, not bugs to be fixed quickly.

  5. The field pivoted: After GPT-3, everyone asked “How do we scale?” instead of “What architecture is best?” Scaling became the primary research lever.

Questions to Guide Your Reading

As you explore these resources, ask yourself:

  1. In-context learning: How does the transformer’s attention mechanism enable the model to learn from prompt examples?

  2. Scale laws: Is there a mathematical relationship between model size, data size, and performance? (Answer: yes, and it’s surprisingly smooth.)

  3. Emergent abilities: Why can GPT-3 do arithmetic and code generation when it was never trained on those specific tasks?

  4. Hallucination: Is hallucination a fundamental limit of transformers, or can it be fixed with better training or architecture?

  5. Alignment: How do we ensure large language models are helpful, harmless, and honest?

Where to Find Pre-prints and Datasets

Final Note

GPT-3 was a watershed moment in AI. Understanding it deeply—its architecture, its capabilities, its limitations—is essential for anyone working in modern AI. The field is moving fast, but the insights from GPT-3 remain foundational.

Good luck with your learning!

← Back to paper overview