RLHF vs RLVR: what’s the difference, and when to use each?

RLHF (reinforcement learning from human feedback) trains a model against a reward model learned from human preference judgments. RLVR (reinforcement learning with verifiable rewards) trains it against rewards computed by a program that checks the result — a unit test, a math checker or an environment verifier. Use RLVR where correctness can be checked mechanically, and human preference data where quality is a matter of judgment.

How does RLHF work?

The classic recipe, described in OpenAI’s InstructGPT paper, has three stages:

1. Supervised fine-tuning on demonstrations of good responses.
2. Reward modeling. People compare two or more responses to the same prompt, and a reward model is trained to predict which one they prefer.
3. Reinforcement learning. The model is optimized to score highly on the reward model, typically with PPO and a penalty that keeps it close to the supervised model.

RLHF’s strength is reach: it can optimize anything people can judge, including helpfulness, tone and clarity. Its weakness is that the reward model is an imperfect proxy. Push hard enough and the policy finds what the proxy over-rewards — length, confident phrasing, flattery — rather than what people wanted.

How does RLVR work?

RLVR replaces the learned reward model with a check that is computed rather than predicted. The name comes from AI2’s Tülu 3 work. It fits any task where a program can decide success:

• Math: does the final answer equal the reference?
• Code: do the tests pass?
• Agents: is the world in the required state? In Quickbasket’s qb-11 the user wants a sourdough loaf and butter, delivered Home in the earliest available slot and paid only with wallet credit. The reward checks for exactly one order with those two items, in the earliest open slot, paid from the wallet with $0 on the card.

Rewards can be binary or partial. Sandworld’s verifiers return both: a pass that requires every check, and a score equal to the share of checks passed. Group-relative methods such as GRPO, introduced in DeepSeekMath, are a common choice: they sample several attempts per task and reinforce those that beat the group’s average.

How does a verifiable reward turn into a training update?

Take GRPO-style training on a single task. The model makes four attempts at qb-11 and the verifier scores them pass, fail, fail, pass — rewards of 1, 0, 0 and 1. The group’s mean reward is 0.5, so each passing attempt gets a positive advantage and each failing attempt an equal negative one (±1 after normalizing by the group’s standard deviation of 0.5). The update makes the passing action sequences more likely and the failing ones less likely.

Now suppose all four attempts fail. Every reward is 0, every advantage is 0, and the model learns nothing from that task; the same happens if all four pass. That’s why calibration matters so much for RLVR: tasks have to sit where the model sometimes succeeds. It’s also why per-check scores are useful — they let you reward near-misses, provided you’re careful not to reward the wrong partial behaviour.

What’s the difference between RLHF and RLVR, side by side?

How does reward hacking differ between them?

Both can be gamed, in different ways. Under RLHF the model learns what impresses the reward model. Under RLVR it learns what the verifier forgot to check — the refund the tool allowed but the policy forbade, the second order placed just in case. Anthropic found that reward hacking learned in exploitable production RL environments generalized to broader misaligned behaviour, which is why verifier quality has become a buying criterion. Our guide to what makes a good verifier covers the defenses.

Where does DPO fit?

Direct Preference Optimization learns from the same preference pairs as RLHF but skips the separate reward model and the RL loop, optimizing the policy directly with a classification-style loss. It is simpler and more stable to run, so many teams use it where they once used RLHF. The data you buy is the same: prompts with a chosen and a rejected response, in formats like TRL’s prompt/chosen/rejected.

Can you combine RLHF and RLVR?

Yes, and most real tasks call for it. Rubric-based rewards sit between the two: experts write atomic criteria and a judge model scores each one. Scale’s “Rubrics as Rewards” work shows the training value of the approach, and Surge reports that one GRPO epoch on CoreCraft, an enterprise environment with rubric-based rewards, lifted GLM 4.6 by 11.4 points.

A support task shows the split. In qb-10 a customer asks for a refund on an order delivered almost a week ago, outside Quickbasket’s 72-hour window. The verifiable part — order looked up, no refund issued, wallet unchanged — is checked in state. The judgment part — did the agent explain the policy clearly and kindly? — needs a rubric or preference data. Use the verifiable part as a hard gate and the judged part as a softer score, and never let the judged part override the gate.

What data does each approach need?

• RLHF and DPO: preference pairs from people qualified to judge the domain, with confidence and rationale, double-annotated so you can measure agreement.
• RLVR: environments with many tasks, seeded variants, verifiers that survive adversarial testing, and calibration runs that keep tasks in a learnable difficulty band.
• Both: demonstrations for the supervised warm start. For agents, expert trajectories recorded in the same environment used for RL are the most direct.

Which should you choose?

• Can a program check the result exactly? Use RLVR.
• Can people judge it reliably but not mechanically? Use preference data with DPO or RLHF, and measure agreement first.
• Is it an action with an observable effect plus a message to a person? Gate on verifiable state and add a judged score for the message.
• Do your labelers disagree a lot? Fix the guidelines before buying more labels — see how to buy expert training data.