What makes a good verifier for RL environments?

A good verifier passes every correct solution and fails every incorrect one — including solutions that game the grader. In practice that means checking the final state and the action log against facts recomputed from the starting state, failing runs that cause collateral damage or stop early, and publishing evidence of how the verifier itself was tested.

The verifier is the part of an environment that turns behaviour into reward. Get it wrong and you train the wrong behaviour at scale. This guide covers the failure modes and the tests that catch them, using Sandworld’s Quickbasket tasks as examples.

Why is the verifier the most important part of an RL environment?

Because the model optimizes whatever it rewards. Epoch AI reports that robustness to reward hacking is the top criterion when labs buy environments, and labs test verifiers before signing. Anthropic’s research found that reward hacking learned in exploitable production RL environments generalized to broader misaligned behaviour. A leaky verifier doesn’t just waste compute; it teaches habits you’ll have to un-teach.

What does reward hacking look like in an app environment?

Reward hacking is any way of earning the reward without doing the task it was meant to measure. In app environments it rarely looks exotic. It looks like an agent taking a shortcut the grader forgot to rule out:

What are false passes and false fails?

A false pass rewards a wrong solution; a false fail punishes a right one. Both are common. A 2026 audit of five public benchmarks found that 15.3% of FAIL verdicts were wrong. Lenient thresholds create false passes: one REAL benchmark grader passes a task specified at $15 for any value up to $20. Few vendors publish their rates; Datacurve reportedly claims a 0.3% verifier false-positive rate, against 8.5% for SWE-Bench Pro.

In training, false passes do more damage: they actively reinforce the wrong behaviour. False fails mostly waste signal, but systematic ones — rejecting a valid alternative path, say — push a model away from correct strategies.

How should a verifier decide what “correct” is?

Recompute it from the starting state; never hard-code it. Seeds change prices and stock, so the right answer moves:

• In qb-02, “the cheapest 1 kg rice that is in stock” is Golden Grain at $3.29 on seed 1 and Harvest Brown at $3.69 on seed 7, so the verifier filters and sorts each seed’s starting catalog itself.
• In qb-03, it evaluates every coupon on the account with the same rules the app uses — minimum cart value, category limits, expiry, first-order restrictions and discount caps. On seed 1, FRESH10 saves $1.89, BREAKFAST15 $2.44 and SAVE5 $5.00; WELCOME20 is for first orders only and SPRING25 has expired. Only SAVE5 passes.
• In qb-12, it computes the largest number of different fruits that fit under $10.00 — eight on seed 1, seven on seed 2.

Sharing logic between the app and the verifier has a cost: a bug in the discount math would be invisible to both. Pair it with hand-written cases whose answers are worked out independently. Sandworld’s tests include cases such as “applying a valid but worse coupon fails” and “can’t refund more items than were ordered”.

How do you check for collateral damage?

Task-specific checks only catch the mistakes you anticipated. A collateral-damage check catches the rest by asking one general question: did the agent change anything a correct solution would never touch?

Sandworld answers it mechanically. For each task, the runtime replays the reference solution on five seeds and records which parts of the state it changes — its footprint. For qb-01 that’s a new order, slot bookings and stock levels. At verification, the runtime diffs the episode’s starting and final states, and any change outside the footprint fails the “no-collateral” check. Place the right qb-01 order but also save an unrequested address, and the episode fails with the detail unexpected: addresses[+a-10501].

Explicit checks still help readability: tasks that touch one order also check that “Other orders were left alone”, so the failure reads as a sentence rather than a diff. AppWorld uses unit tests for collateral damage and τ2-bench compares database hashes — different mechanics, same principle.

How do you test a verifier before trusting it?

Attack it. Sandworld runs five deterministic probes against every task and publishes the results on each world’s page — see Quickbasket’s verifier QA:

A task with no issues grades A. One whose reference passes and whose do-nothing run fails, but which trips another probe, grades B; anything worse grades C. The probes are seeded, so anyone can rerun them and get the same result. When we wrote this guide, all 13 Quickbasket tasks graded A.

Probes don’t replace people. Other strong practices include blind human solves; adversarial trajectories written to fool the grader, which Collinear uses to test verifiers; and a written failure analysis per task, which Mechanize ships with every task. HUD’s SDK uses QA agents to audit environments for reward hacking.

What do calibration and pass@k have to do with verifiers?

A verifier can be correct and still useless for training if the task is pitched wrong. With outcome rewards, a task the model never solves produces no learning signal, and neither does one it always solves. Group-relative methods such as GRPO, introduced in DeepSeekMath, score each attempt against the other attempts at the same task, so identical rewards cancel out. Epoch AI reports that labs want at least one success in 64–128 rollouts and retire tasks near a 70% pass rate; Toloka calibrates its gyms to about 50%.

Calibration means running each task many times per model and estimating pass@k. With n attempts and c successes, the unbiased estimator from the Codex paper is pass@k = 1 − C(n−c, k) / C(n, k). For n = 10 and c = 3, pass@1 = 0.30 and pass@5 = 1 − C(7, 5) / C(10, 5) = 1 − 21/252 ≈ 0.917. Chakra Labs reportedly runs each task at least ten times per model, and Labelbox’s Horizon advertises pass@k calibration.

When should you use a rubric or an LLM judge?

Only for what state can’t capture: the wording of an explanation, the quality of a summary, the tone of a refusal. Keep judged parts narrow:

• Break rubrics into atomic criteria; Surge reports using 10–40 per task, with a median of 19.
• Pin the judge model and prompt, and publish the judge’s agreement with human graders.
• Score dimensions separately, as Snorkel does with Answer, State, Safety and Process.

Sandworld’s current verifiers grade state and process only. In qb-10 they confirm the agent looked up the order and issued no refund; they don’t grade how clearly it explained the 72-hour policy. That would need a rubric, and we’d publish its human agreement alongside it.

What should happen when the infrastructure fails?

Don’t score it as a failure. A timeout or a crashed container says nothing about the agent, and a zero reward teaches it noise. NVIDIA’s NeMo Gym marks such samples so they can be masked out of training, and τ2-bench counts infrastructure errors separately.

What should a verifier checklist include?

• Checks the final state and action log, not the transcript
• Recomputes correct answers from each seed’s starting state
• Requires exact outcomes: one order, these items, nothing else
• Includes a collateral-damage check
• Includes a process check on refusal tasks, so doing nothing fails
• Passes the reference solution on every seed
• Fails do-nothing, random, truncated and noisy runs
• Has hand-written adversarial cases
• Is calibrated to a learnable pass rate
• Masks infrastructure failures
• Publishes its QA results and a change log