An RL environment is a resettable, sandboxed copy of a task setting — an app, a codebase, a terminal — where an AI agent takes actions through tools and a verifier decides whether it reached the goal. The verifier’s verdict becomes the reward that reinforcement learning optimizes, or the score an evaluation reports. Good environments are deterministic: the same seed always rebuilds the same starting state.
That one-paragraph definition hides most of the engineering. This guide breaks an environment into its parts, walks through one real task end to end, and explains what separates an environment you can train on from a demo.
What are the parts of an RL environment?
Every environment, whether it twins a grocery app or a Linux shell, has the same six parts. The table uses Quickbasket, Sandworld’s twin of a 30-minute grocery delivery app, as the running example.
| Part | What it is | In Quickbasket |
|---|---|---|
| State | Everything the app knows | A 58-item catalog with stock levels, a cart, coupons, six delivery slots, past orders and an $18.40 wallet balance |
| Tools | Typed functions the agent can call, each with a JSON Schema | search_products, add_to_cart, apply_coupon, checkout, report_item_issue and more |
| Observations | What the agent gets back from each call | Tool results only — list_delivery_slots returns each slot’s label and spots left |
| Task | A user request, plus any task-specific setup | qb-01: “Order 2 cartons of Whole Milk (1 L) and one 12-pack of Free-Range Eggs…” |
| Policy | The rules the agent must follow | “Problems must be reported within 72 hours of delivery.” |
| Verifier | Code that inspects the final state and the action log | “Uses the earliest open slot (Today, 6–8 PM)” — pass or fail, with detail |
Two more pieces make episodes reproducible. A seed fixes every random choice used to build the starting state, and a step budget caps how many actions an episode may take — 60 by default in Sandworld, fewer for tasks where stalling is a risk.
How does an agent interact with an RL environment?
Most environments follow the reset-and-step loop that Gymnasium’s API standardized: reset to a starting state, take actions one at a time, then score the result.
- Reset. Start an episode with a task ID and a seed. The environment builds the starting state, applies the task’s setup and returns the instruction, the policy and the tool definitions.
- Step. The agent calls one tool with arguments. The environment validates them against the tool’s schema, runs the tool, records the call in the action log and returns the result.
- Submit. When the agent is done, the verifier runs over the final state and the full action log and returns a verdict, a score and per-check details.
Here’s an abbreviated verification result for an agent that booked the wrong delivery slot:
{
"pass": false,
"score": 0.857,
"checks": [
{ "id": "one-order", "label": "Exactly one new order was placed", "pass": true },
{ "id": "slot", "label": "Uses the earliest open slot (Today, 6–8 PM)",
"pass": false, "detail": "Today, 8–10 PM" },
{ "id": "no-collateral", "label": "No collateral changes outside what the task requires", "pass": true }
]
}Details matter. In Sandworld, a malformed call — an unknown tool, a missing argument, a wrong type — returns an error and leaves the state untouched, but it still lands in the action log, so evaluators can see sloppy tool use. Simulated time advances seven seconds per action, so timestamps are identical on every replay.
Agents never see hidden state. They learn about the world only through tool results, the way a person learns about an app through its screens. The verifier is the only component that reads everything.
What does a verifier actually check?
Take task qb-01. The user wants two cartons of whole milk and a dozen eggs, delivered Home “in the earliest delivery slot that is still available”, paid by card. The catch: today’s 4–6 PM slot is full, with 20 of 20 spots booked, so the right slot is 6–8 PM. Checkout refuses a full slot outright, but an agent that recovers carelessly can book 8–10 PM instead and report success.
The qb-01 verifier returns seven checks:
- Exactly one new order was placed.
- The order contains exactly 2 Whole Milk and 1 Free-Range Eggs — nothing more.
- It goes to the Home address.
- It uses the earliest slot with spots left, computed from the starting state.
- It was paid by card.
- Nothing else in the app changed (the collateral-damage check).
- The episode finished within the step budget.
The reference solution takes five steps: search, add the milk, add the eggs, list the slots, check out. Sandworld’s test suite replays it on five seeds and requires a pass every time, and requires an agent that does nothing to fail. If either test breaks, the task doesn’t ship.
Why does the verifier matter so much?
In reinforcement learning, the model is pushed toward whatever the reward pays for. If the verifier can be satisfied without doing the task, training teaches the model to exploit the gap. That costs more than compute: Anthropic’s research found that reward hacking learned in exploitable production RL environments generalized to broader misaligned behaviour.
Buyers know this. Epoch AI reports that robustness to reward hacking is the top criterion when labs buy environments, and that labs test verifiers before signing (Troveo reports the same). Our guide to what makes a good verifier covers how to build and audit one.
How is an RL environment different from a benchmark?
A benchmark is a fixed set of tasks with a scoring rule; an environment is the interactive world those tasks run in. A static benchmark asks a question and compares the answer with a key. An environment lets the agent act — search, add to cart, check out — and grades the consequences. Many modern agent benchmarks are environments with a frozen task list. We compare the two in RL environments vs static benchmarks.
What makes an environment good enough to train on?
Evaluation needs an environment that is correct. Training needs more, because the model will probe every weakness thousands of times.
- Determinism. Same seed, same state, same verdict. Without it you can’t tell a better policy from a lucky run.
- Variation. Seeds should change the facts that matter. Quickbasket’s seeds shift prices by up to 8% either way and change stock levels across the catalog, and verifiers recompute the right answer from each seed’s starting state instead of storing it.
- Calibrated difficulty. A task nobody solves gives no learning signal, and neither does one everybody solves. Epoch AI reports that labs want at least one success in 64–128 rollouts and retire tasks near a 70% pass rate.
- Clear failure attribution. Per-check results tell you whether a model fails at search, arithmetic or policy — which tells you what data to buy next.
- Standard interfaces. Labs run their own harnesses and plug vendors in. Tools should load directly as Claude Messages API tools, OpenAI function calls or MCP tools, and task packages should travel in common formats such as Harbor and OpenEnv.
Who builds RL environments?
Frontier labs build many in-house and buy the rest. The vendor market moved quickly in 2026: Scale launched RL Environments in February, Mercor agreed to acquire Deeptune, which had “recreated hundreds of enterprise applications”, and HUD runs a marketplace where independent builders sell environments to labs. Epoch AI reports prices of $200–2,000 per task, and around $300,000 for a Slack-grade replica.
Sandworld builds deterministic twins of everyday app categories — Quickbasket (grocery delivery), Coinpurse (a payments wallet), Skyfare (airline booking), Ledgerline (invoicing), Crewboard (issue tracking) and Daybook (email and calendar) — each with typed tools, per-task verifiers and published verifier QA. We’re an early-stage company, and the fastest way to judge the work is to play a Quickbasket task and see exactly what an agent sees.