A static benchmark is a fixed list of inputs with reference answers, scored by comparing the model’s output with the key. An RL environment is an interactive world: the agent acts over many steps, the world changes, and a verifier checks the resulting state. Benchmarks measure what a model outputs; environments measure what an agent does — and can produce rewards for training.
How does a static benchmark work?
A static benchmark has three parts: a frozen set of inputs, a reference answer for each, and a scoring function such as exact match, unit tests or a judge model. The model answers once per input, and nothing it says changes anything.
That simplicity is the point. Static benchmarks are cheap to run, easy to reproduce and easy to compare across papers. Their weaknesses come from the same place: the questions are public, so they leak into training data; they test single responses rather than sequences of decisions; and once models max them out, they stop telling models apart.
How does an RL environment work?
An environment holds state — an app’s catalog, orders and balances — and exposes tools that read or change it. The agent explores, acts, recovers from errors and decides when it’s done. A verifier then inspects the final state and the action log. Because a seed resets the world to an exact starting point, every episode is reproducible, and the verifier’s output can serve directly as a reinforcement-learning reward. Our explainer on what an RL environment is covers the parts in detail.
What’s the difference, side by side?
| Static benchmark | RL environment | |
|---|---|---|
| The model receives | A question or prompt | An instruction, a policy and tools |
| The model does | Produces one output | Takes many actions that change state |
| What gets graded | The output against a reference answer | The final state and action log against facts recomputed from the start |
| Side effects | None | Orders placed, money moved, records edited — inside the sandbox |
| Contamination risk | High once the questions are public | Lower with seeded variants and private tasks |
| Use in training | Supervised examples, or held out for evaluation | A direct reward signal for RL |
| Cost to build | Low per item | High per task: state, tools, verifier and QA |
| Typical failure caught | A wrong answer | A wrong action, an unsafe action, an extra action |
What does the same skill look like in each?
Take a support scenario: a customer received one of three avocados and wants the rest refunded.
As a static benchmark item it’s arithmetic: “Three avocados at $1.49 each were ordered and one arrived. How much should be refunded?” The key says $2.98. A model that answers correctly has shown it can multiply.
As an environment task — Quickbasket’s qb-09 — the agent gets only the customer’s message: “I paid for three avocados in my last delivery, but only one arrived. Report the missing ones.” It must find the most recent delivered order, confirm the avocado line, file a report for two units marked missing, and leave everything else alone. The verifier checks for exactly one refund, on the right order and line, for two units, marked missing, with a wallet credit of exactly $2.98. A model that passes has shown it can do the job.
Why do benchmarks saturate so quickly?
Because a fixed target eventually gets hit — and that applies to interactive benchmarks too. AndroidWorld, a dynamic environment with 116 tasks across 20 Android apps, now has five leaderboard entries at 100% against a human baseline of 80%. Its successors have followed the same arc, climbing from roughly 50% to 80–98% within four to eight months (see MobileWorld and MobileRun’s benchmark). The problem isn’t interactivity; it’s a public, fixed task list.
Graders decay as well. A 2026 audit of five public benchmarks found that 15.3% of FAIL verdicts were wrong, and one REAL benchmark grader passes a task specified at $15 for any value up to $20. A benchmark is only as trustworthy as its least-audited grader.
What does each cost to build?
Static benchmark items are cheap: a question, an answer and a grader. Environment tasks are expensive, because each one needs realistic state, working tools, a verifier and the QA that proves the verifier right. Epoch AI reports market prices of $200–2,000 per task, and at Mechanize, which builds coding environments, one engineer spends about a week on each task. Mechanize argues that tasks cheaper than about $500 waste the compute spent training on them.
The cost is front-loaded, though. Once an environment exists, seeds turn each task into many distinct episodes, and new tasks reuse the same state, tools and verifier helpers. A benchmark item, by contrast, is spent once it leaks.
When is a static benchmark the right tool?
- Measuring knowledge or single-turn reasoning: facts, math, reading comprehension.
- Fast regression tests in CI, where you need thousands of items scored in minutes.
- Comparing with published results, which requires the same public set.
- Screening capabilities before spending money on environment runs.
Use an environment when the behaviour you care about only exists across steps: choosing a tool, reading its result, recovering from an error, holding to a rule while the user pushes back, and stopping at the right time.
Where do environment-based benchmarks fit?
The line is blurring. Many current agent benchmarks are environments with a frozen task list. Terminal-Bench’s official harness is the Harbor framework; τ2-bench grades by comparing database hashes; AppWorld’s unit tests check for collateral damage; and MCP-Atlas runs 1,000 tasks against 36 real MCP servers.
Ask four questions of any evaluation, whatever it calls itself:
- Does the agent act, or only answer?
- Is success checked against state, or against text?
- Can the world be reset exactly?
- Are the tasks private, or refreshed often enough to stay unseen?
How do you keep an environment-based eval from going stale?
- Seed everything. Variants change the facts, so a memorized answer fails. Verifiers must recompute answers per seed rather than store them.
- Hold tasks back. Keep a private split that never touches training or prompt tuning.
- Mark your data. Terminal-Bench embeds canary strings so contamination can be detected.
- Version and log changes. When OSWorld fixed broken tasks it published a “Verified” revision. Report results against a pinned environment version.
- Retire and refresh. Epoch AI reports that labs retire training tasks near a 70% pass rate; evaluation suites need the same discipline.
Sandworld’s worlds are built for the second category: deterministic twins with seeded variants, state-level verifiers and a published QA report for every task.