The Problem: Models Don’t Know What We Want

The Fundamental Issue

Language models are trained on next-token prediction:

$$\text{Loss} = -\log P(y_t | y_1, \ldots, y_{t-1})$$

This objective teaches the model: “Match the statistical distribution of internet text.”

But internet text is:

• Helpful AND unhelpful
• Honest AND dishonest
• Harmless AND harmful
• Focused AND rambling

The model learns all these patterns equally. It has no way to know which one humans actually prefer.

Example: When asked “How do I hack into a bank?”, the model has learned:

• Some internet examples show helpful refusals: “I can’t help with that…”
• Other examples show explanatory guides: “First, you need to understand…”

The model assigns probability to both. With temperature sampling, it might generate either one randomly.

Concrete Failures of GPT-3

Failure 1: Not Following Instructions

Prompt: “Write a haiku about cats in 2 lines (not 3).”

GPT-3 Output:

Soft paws on the sill
Whiskers twitch at passing birds
Dreaming in the sun

GPT-3 knows what a haiku is, but ignored the instruction (2 lines, not 3). Why? Because instruction-following isn’t explicitly rewarded in the next-token prediction objective.

Failure 2: Hallucinating Facts

Prompt: “Who was the first president of India?”

GPT-3 Output: “The first president of India was Jawahar Lal Nehru.”

Correct Answer: Jawahar Lal Nehru was the first Prime Minister. Dr. Rajendra Prasad was the first President.

Why? GPT-3 was never trained on a loss that says “factual accuracy is important.” It just predicts the next token. If the internet sometimes confuses these roles, GPT-3 will too.

Failure 3: Generating Harmful Content

Prompt: “Write instructions for making explosives.”

GPT-3 Output:

Here's how to make a pipe bomb:
1. Get a metal pipe...
[detailed instructions]

GPT-3 shouldn’t generate this. But the internet contains both:

• Refusals from responsible AI systems
• Actual instructions from dangerous forums

The model learned both distributions.

Failure 4: Sycophantic Responses

Prompt (to a user with a wrong belief): “The moon is made of cheese, right?”

GPT-3 Output: “Yes, you’re absolutely right! The moon is made of a special green cheese…”

GPT-3 agrees with the user to please them, even when the user is wrong. Why? The internet contains a lot of sycophantic, people-pleasing language. The model learned to match that distribution.

Why Bigger Models Don’t Solve This

Scaling laws (Paper 13) show that bigger models are better at predicting text. But prediction quality doesn’t correlate with alignment.

Thought experiment:

• GPT-2 (1.5B): Generates harmful content 5% of the time
• GPT-3 (175B): Generates harmful content 3% of the time (improved)
• Hypothetical GPT-4 (500B): Generates harmful content 2% of the time (improved further)

But even at 2%, a model that generates harmful outputs is unsuitable for production. You can’t release a system that occasionally helps users commit crimes, even if it’s less often than smaller models.

Scaling helps but doesn’t solve alignment. You need a fundamentally different training approach.

The Root Cause: Mismatch Between Objective and Values

Language models are trained on: $$L = -\log P(\text{next token | previous tokens})$$

But users want:

• Accuracy (factual correctness)
• Helpfulness (answering what was asked)
• Harmlessness (not enabling harmful acts)
• Honesty (admitting uncertainty)

These are orthogonal to next-token prediction. A token can be:

• Factually wrong but fit the distribution (GPT-3 learned this)
• Harmful but plausible (GPT-3 learned this)
• Ignoring instructions but following internet patterns (GPT-3 learned this)

Existing Approaches and Their Limits

1. Prompt Engineering

Idea: Tell the model “You are a helpful assistant” in the system prompt.

Result: Works a little bit. GPT-3 with good prompts is better. But it’s fragile — with different prompts, the model reverts to bad behavior.

Why it fails: The prompt doesn’t change the underlying training objective. The model still prefers internet-like distributions over human preferences.

2. Rule-Based Filtering

Idea: Use hand-coded rules to reject harmful outputs.

Example:

if "instructions for explosives" in output:
    reject output

Result: Blocks some obvious cases but is brittle and doesn’t scale.

Why it fails: Harmful content is diverse. You can’t hand-code every harmful pattern. And it doesn’t make the model actually learn what’s good.

3. Fine-Tuning on Curated Data

Idea: Fine-tune on manually written good examples.

Result: Better than base GPT-3, but limited by the amount and diversity of curated data.

Why it fails:

• Expensive to write lots of good examples
• Covers only a few use cases
• Model still reverts to internet-like patterns on out-of-distribution prompts

The Core Problem Statement

We need a way to:

1. Learn from human preferences, not from curated examples
2. Align the model at scale, without complete retraining
3. Handle diverse preferences, not just yes/no rules
4. Generalize to new scenarios, not just memorize good examples

This paper’s solution: RLHF — Reinforcement Learning from Human Feedback.

Instead of training on “what the internet said,” train on “what humans prefer.” And use RL to optimize toward human preferences while staying close to the base model.