What Is RLHF? Reinforcement Learning Explained

RLHF, short for reinforcement learning from human feedback, is a training technique that shapes a language model’s outputs to match what humans actually judge as helpful, honest, and appropriate — rather than simply what’s statistically likely to follow a prompt. It’s the step that turns a raw pretrained model, which is good at predicting plausible next tokens, into something that behaves more like a cooperative assistant.

Why pretraining alone isn’t enough

A model trained only on next-token prediction over a large text corpus, as covered in what an LLM is, learns the statistical patterns of language extremely well, but it has no built-in notion of what a good response looks like. Asked a question, a purely pretrained model might complete it with another question, ramble into unrelated territory, or reproduce something harmful it saw in training data — all of which are perfectly plausible continuations of text, just not what a user wants. RLHF exists to close that gap between “plausible” and “helpful.”

The three-stage pipeline

RLHF is usually described as building on top of an already pretrained, and often already instruction-tuned, base model:

1. Supervised fine-tuning (SFT). Human annotators write example responses to a range of prompts, demonstrating the tone, format, and behavior the model should produce. The base model is fine-tuned on this dataset the same way described in what fine-tuning is — continuing training on a smaller, curated dataset rather than starting from scratch.
2. Reward model training. Instead of writing full responses, annotators are shown several model outputs for the same prompt and rank them from best to worst. A separate model — the reward model — is trained to predict that ranking, effectively learning a scoring function that estimates how much a human would like any given response.
3. Reinforcement learning optimization. The original language model is then fine-tuned using reinforcement learning, where the reward model’s score acts as the reward signal. The model generates a response, the reward model scores it, and the language model’s parameters are nudged to make higher-scoring responses more likely in the future. This step commonly includes a penalty that keeps the fine-tuned model from drifting too far from its starting point, so it doesn’t sacrifice general language ability while chasing reward.

The result is a model that’s still built on the same underlying architecture — see what a transformer is for the mechanics of that foundation — but whose output distribution has been reshaped to favor responses that align with the preferences captured in the reward model.

Why a separate reward model, instead of asking humans directly

Reinforcement learning typically requires many thousands of feedback signals to converge, far more than it’s practical to collect from humans in real time during training. Training a reward model once, from a comparatively small set of human comparisons, and then using that model to generate a reward signal at scale is what makes the reinforcement learning stage tractable. The tradeoff is that the reward model is itself an approximation of human judgment — it can be gamed by the language model finding outputs that score well on the reward model without actually being what a person would prefer, a failure mode generally called reward hacking. This is part of why the optimization step includes a constraint keeping the model close to its pre-RL behavior.

What the reward model is actually scoring

The quality of an RLHF-trained model is bounded by the quality of the reward model, which is itself bounded by the quality and consistency of the human comparisons it was trained on. Annotators don’t always agree with each other on which of two responses is better, particularly for subjective qualities like tone or for prompts touching contested topics, and that disagreement becomes noise the reward model has to average over. This is one reason organizations doing RLHF invest heavily in annotation guidelines and annotator training — the reward signal downstream is only as reliable as the human judgments feeding it upstream, and inconsistent guidance produces a reward model that has learned an inconsistent notion of “better.”

RLHF vs plain fine-tuning

	Supervised fine-tuning	RLHF
Training signal	Exact target outputs written by humans	A learned reward score comparing outputs
What it teaches	Mimicking demonstrated examples	Preferring outputs humans rank higher, including ones never explicitly written
Data needed	Full example responses	Comparisons/rankings between existing responses
Typical use	Teaching format, tone, task-following	Aligning behavior with nuanced preferences (helpfulness, safety, honesty)

Supervised fine-tuning teaches a model to imitate; RLHF teaches it to be evaluated favorably, which lets it generalize preference beyond the exact examples annotators wrote by hand.

Where RLHF fits alongside other techniques

RLHF is one way to align a model with human preferences, but not the only one — newer preference-optimization methods aim to reach similar outcomes without a separate reinforcement learning loop, training directly on preference comparisons instead. RLHF is also distinct from techniques that change what information a model has access to at inference time, like retrieval-augmented generation, which supplies external facts rather than reshaping the model’s behavior. And it’s separate from prompting a model to reason step by step, which reasoning models build on more directly — RLHF is about training-time alignment of how a model responds, not about the inference-time strategy it uses to arrive at an answer.

The takeaway

RLHF trains a language model to prefer responses that score well against a learned model of human judgment, built from a comparatively small set of human comparisons between candidate outputs. The pipeline runs supervised fine-tuning first to establish baseline behavior, trains a reward model on human rankings, then uses reinforcement learning to nudge the language model toward higher-scoring responses while staying anchored to its original behavior. It’s the training step most responsible for the gap between a raw pretrained model and one that behaves like a usable assistant.