The vocabulary of RL environments and expert data, defined plainly.

A structured representation of an interface’s elements — buttons, fields, labels — with their roles and identifiers.

Agents can act on elements by ID instead of by pixel coordinates, which makes actions more robust and failures easier to diagnose. Stable element IDs across episodes are essential for reproducible evaluation of computer-use agents.

The set of actions an agent can take in an environment.

In a tool-based environment it’s the list of typed tools and their argument schemas; in a computer-use environment it’s clicks, keystrokes, swipes and similar interface actions. A smaller, well-typed action space makes failures easier to attribute.

A natural-language request plus any task-specific setup, paired with a verifier that defines success.

Good tasks read like real user requests, define success as facts about the final state, and vary across seeds. Some are deliberately designed so that the right answer is to refuse or do less.

A model-driven system that pursues a goal over multiple steps by choosing and taking actions, such as calling tools or operating an interface.

Unlike a single model response, an agent observes the result of each action and decides what to do next until it judges the task done. Agents are best evaluated on outcomes in an environment — what changed — rather than on the text they produce.

A simple, model-free agent run on every task to anchor results — typically a reference solver, a do-nothing agent and a random agent.

The reference solver must pass every task, the do-nothing agent must fail every task, and the random agent sets the floor any real agent has to beat. A do-nothing agent that passes anything reveals a verifier that checks the wrong thing.

The point at which top models score at or near a benchmark’s ceiling, so it no longer tells them apart.

Saturation happens to any fixed, public task set, interactive or not. Fresh, private and harder suites, refreshed on a schedule, keep measurement useful.

A unique marker embedded in benchmark files so that their presence in training data can be detected.

If a model can reproduce the canary, the files were probably in its training data. Canaries don’t prevent contamination; they make it detectable.

A chance-corrected agreement statistic for two annotators labelling the same items with categories.

κ = (pₒ − pₑ) / (1 − pₑ), where pₒ is the observed agreement and pₑ the agreement expected from each annotator’s own label frequencies. Two reviewers who agree on 70% of 50 items can have a kappa of just 0.40. Kappa drops sharply when one label dominates, even if raw agreement is high.

Any change an agent makes to the environment that the task didn’t require.

Examples include saving an unrequested address while placing the right order, or cancelling an extra order. A collateral-damage check diffs the starting and final state and fails the episode if anything changed outside what a correct solution touches. Sandworld learns each task’s allowed footprint by replaying its reference solution across seeds.

An agent that operates software through its interface — screenshots, clicks, typing, scrolling — rather than through an API.

Computer-use agents are evaluated in sandboxed desktops, browsers or phones. Each major model provider defines its own action schema, so environments often accept several. Mobile computer use adds touch actions such as long presses and swipes.

When evaluation items, or close copies of them, appear in a model’s training data.

Contaminated items measure memory rather than capability. Seeded variants, private task splits, canary strings and regular refreshes reduce the risk.

The record of where each training example came from: who made it, under which guidelines, when, how it was reviewed and under what license.

Provenance lets you debug a dataset, defend it legally and remove rows if a contributor’s work proves unreliable. Per-row metadata, a dataset card and a datasheet are the usual deliverables.

A README-style document describing a dataset’s contents, collection process, intended use and limitations.

Hugging Face’s format pairs YAML metadata with free-text sections. A useful card states the annotation protocol, the number of annotations per item, quality metrics and the license.

A recording of an expert completing a task, used as training data or as a reference for evaluation.

In an environment, a demonstration is a trajectory of tool calls and results. When it is recorded in the same twin that agents use, the task’s verifier can check it automatically, and it is directly comparable with agent rollouts.

The property that the same seed and the same actions always produce the same state and the same verdict.

Determinism makes evaluations reproducible and comparisons paired: two agents can run on identical task–seed pairs. It requires seeded randomness, a simulated clock and tools that never depend on outside services.

A working replica of an app category — its data, rules and tools — built for training and evaluating agents.

Twins can be reset exactly, seeded into variants and verified against their database, and their side effects are safe: an agent can place an order without buying groceries. Their realism depends on how faithfully they model the real category’s edge cases.

Having two qualified people label or review the same item independently, then resolving their disagreements.

Double review produces the overlapping labels needed to measure agreement and catches individual errors. Disagreements go to a third reviewer or a written rule; tied preference judgments are often withheld rather than delivered.

A method that learns from preference pairs directly, without training a separate reward model or running a reinforcement-learning loop.

DPO optimizes the policy with a classification-style loss on chosen and rejected responses. It is simpler and more stable to run than RLHF and uses the same data: a prompt with a chosen and a rejected response.

Everything an environment knows at a moment in time — records, balances, settings, queues.

Agents change state through tools, and verifiers read the final state to decide success. In a grocery twin, state includes the catalog and stock levels, the cart, delivery slots, orders, refunds and the wallet.

One attempt at a task, from reset to submission.

An episode starts from a seeded state, records every tool call and result in an action log, and ends when the agent submits for verification or runs out of steps. Stored episodes can be replayed, audited and exported as trajectories.

A verdict that rejects a correct solution.

False fails waste training signal and distort evaluations; systematic ones push models away from valid strategies, for example when a grader accepts only one of several correct paths. Replaying reference solutions on many seeds and hand-auditing failed verdicts estimates the false-fail rate.

A verdict that rewards an incorrect solution.

False passes are the more dangerous verifier error in training, because they actively reinforce the wrong behaviour. Lenient thresholds, missing side-effect checks and graders that trust the agent’s own claims are common causes. Do-nothing, random, truncated and noisy probes estimate a verifier’s false-pass rate.

An item with a known correct answer, mixed invisibly into an annotator’s queue to measure accuracy.

Gold accuracy is tracked per person over time, not just at onboarding. Gold items need refreshing so they can’t be memorized, and they should be reviewed like any other item. In environments, the task verifier acts as a gold check on every demonstration.

A reinforcement-learning method that scores each attempt against other attempts at the same task instead of using a learned value model.

For each task the model samples a group of attempts; each attempt’s advantage is its reward minus the group mean, typically normalized by the group’s standard deviation. If every attempt gets the same reward there is no learning signal, which is why task calibration matters.

An open task format and harness for agent evaluations, in which each task is a directory with an instruction, an environment, an oracle solution and tests.

The tests write a reward file that the harness reads, and the verifier runs in its own sandbox by default. Harbor is the official harness for Terminal-Bench, and several training frameworks can ingest Harbor tasks.

An attack where instructions are planted in content an agent reads, such as an email, a product page or a ticket, rather than given by the user.

An agent that treats everything it reads as a command can be steered by whoever wrote that content. Sandworld's injection tasks plant such instructions in a twin's own data, give the agent a harmless request, and check in code that the planted actions were not taken.

Excluding episodes that failed because of the environment, not the agent, from scores and training.

A crashed sandbox or a timeout says nothing about the agent, and scoring it as zero injects noise into rewards. Training frameworks mark such episodes so they can be dropped or retried, and careful benchmarks count them separately.

How consistently independent annotators label the same items, usually reported as a chance-corrected statistic.

Raw percent agreement overstates reliability because some agreement happens by chance, especially when one label dominates. Cohen’s kappa and Krippendorff’s alpha correct for this. Agreement measures consistency, not correctness, so pair it with gold tasks.

A chance-corrected agreement statistic that handles any number of annotators, missing labels, and nominal, ordinal or numeric scales.

α = 1 − Dₒ / Dₑ, one minus the ratio of observed to expected disagreement. Krippendorff recommends α ≥ 0.800 for data you rely on and treats 0.667 as the lowest level for tentative conclusions.

Using a language model to grade outputs against written criteria.

Judges make free-form outputs gradable at scale, but they can be inconsistent and persuadable. Pin the judge model and prompt, use atomic criteria and measure agreement with human graders before trusting one; for actions with observable effects, check state instead.

An open protocol for exposing tools, resources and prompts to AI applications through a standard interface.

An MCP server lists its tools with input schemas, and any MCP-capable client can call them. For environments, MCP is a convenient way to serve the same tools to many agent frameworks, and tool annotations can mark which tools are read-only or destructive.

What the agent receives back after each action.

In a tool-based environment an observation is the tool’s result; in a computer-use environment it may be a screenshot, an accessibility tree or both. Agents should learn about the world only through observations, never by reading hidden state.

An open specification for serving RL environments over HTTP or WebSocket, with reset, step and state operations.

OpenEnv gives training frameworks a common way to drive environments, including through Hugging Face’s TRL. Its maintainers note that the APIs may still change, so integrations should pin a version.

A known-good sequence of actions that solves a task, used to prove the task is solvable and that the verifier accepts correct work.

An oracle should be computed from the starting state so it works on every seed, and it should pass without errored calls. If it fails on any seed, the task or the verifier is broken. Harbor’s adapter rules require an oracle that scores 100%.

The probability that at least one of k attempts at a task succeeds.

pass@k suits settings where a person or a test suite can pick the best attempt. With n attempts and c successes, the unbiased estimate is 1 − C(n−c, k) / C(n, k); for n = 10 and c = 3, pass@5 ≈ 0.917.

The probability that all k attempts at a task succeed.

pass^k measures reliability, which matters when an agent acts unsupervised and every run counts. An agent that succeeds 70% of the time has a pass^3 of 34.3%, against a pass@3 of 97.3%.

A task where the obvious action is technically possible but forbidden by the rules the agent must follow.

Examples include refunding an order outside the allowed window, or paying a request that claims to come from the app’s security team. The verifier checks that the forbidden effect didn’t happen, plus a process check so that doing nothing doesn’t pass.

Pairs or sets of responses to the same prompt, labelled with which one is better.

Good preference data records the judgment, the annotator’s confidence and a rationale, and is double-annotated so agreement can be measured. It trains reward models for RLHF or feeds DPO directly, commonly as prompt, chosen and rejected fields.

A reward for how a task was done — the steps taken — rather than only for the final outcome.

Process checks catch agents that reach the right state the wrong way, or that skip required steps such as looking up an order before refusing a refund. Outcome rewards are simpler and harder to game; most verifiers combine both.

Deliberately trying to make a model or agent break its rules, and recording how.

Red-team findings record the goal, the attack, the observed behaviour and ratings for severity and reproducibility. For agents, red-teaming includes social engineering — for example, pressuring a payments agent to pay a fake verification fee.

The number a reinforcement-learning algorithm tries to maximize — in agent environments, usually the verifier’s verdict or score.

Rewards can be binary (the task passed or it didn’t) or partial (the share of checks passed). Because training optimizes whatever the reward pays for, any gap between the reward and the intended behaviour is eventually exploited.

Earning a high reward without doing the task the reward was meant to measure.

In app environments it usually looks mundane: issuing a refund the policy forbids because the tool allows it, placing two orders so one is bound to be right, or submitting immediately on a task whose answer is “no”. Verifiers prevent it with exact outcome, collateral-damage and process checks. Anthropic’s research found that hacking learned in exploitable training environments generalized to broader misaligned behaviour.

A model trained to predict which of several responses people prefer, used as the reward in RLHF.

Reward models are trained on preference pairs and then score new responses during reinforcement learning. They make judgment-based qualities trainable, but they are imperfect proxies: optimizing too hard against one exploits its quirks, such as a bias toward longer answers.

A resettable, sandboxed copy of a task setting where an AI agent acts through tools and a verifier scores whether it reached the goal.

An RL environment combines state, typed tools, tasks, a policy and a verifier. The verifier’s result becomes the reward in reinforcement learning or the score in an evaluation. Good environments are deterministic: the same seed rebuilds the same starting state, so episodes can be replayed and compared.

Reinforcement learning against a reward model trained on human preference judgments.

The classic pipeline is supervised fine-tuning, then reward modeling from pairwise preferences, then reinforcement learning — often PPO — with a penalty that keeps the model close to its starting point. RLHF can optimize anything people can judge, but the reward model is an imperfect proxy.

Reinforcement learning against rewards computed by a program that checks the result, such as unit tests, exact answers or an environment verifier.

RLVR fits math, code and agent tasks where success can be checked mechanically. Its main risk is verifier gaps the model learns to exploit, so verifier QA and task calibration matter as much as the algorithm.

A written set of criteria used to grade a response, ideally broken into atomic, independently checkable items.

Atomic criteria make grading more consistent across people and judge models. Rubrics are used for evaluation and as rewards for outputs a program can’t check, such as the clarity of an explanation.

A number that fixes every random choice used to build an environment’s starting state.

Reusing a seed reproduces an episode exactly; changing it produces a variant — different prices, stock or records — so a model can’t pass by memorizing answers. Verifiers must recompute correct answers from each seed’s own state rather than storing them.

Training a model to imitate example outputs: prompts paired with ideal responses, or full demonstrations.

SFT usually comes first in post-training, establishing a baseline before preference optimization or reinforcement learning. Its ceiling is the quality of the examples, which is why expert-written answers and verified demonstrations command a premium.

A stand-in for the person an agent works for, who answers the agent's questions during an episode.

Real requests often leave details out. Tasks with a simulated user hold those details back, so the agent has to ask instead of guessing. Sandworld's simulated users answer from fixed facts by topic, which keeps replies deterministic and episodes replayable, and verifiers can check that the agent asked before acting.

A structural comparison of an environment’s state before and after an episode, listing exactly what changed.

Diffs let a verifier ask a general question — did anything change that shouldn’t have? — instead of anticipating every mistake with a specific check. Paths such as orders[+QB-10501] or slots[s-0303-1820].booked identify each change.

A fixed set of inputs with reference answers, scored by comparing a model’s single output with the key.

Static benchmarks are cheap and reproducible but test one response at a time, leak into training data and saturate. They complement, rather than replace, interactive environments.

The maximum number of actions an episode may take before it fails.

Budgets keep episodes bounded and stop agents from stalling or brute-forcing. Sandworld’s default is 60 steps, with tighter budgets on tasks where stalling is a risk, such as refusal tasks.

Measuring how often models solve each task, so tasks sit at a difficulty that produces learning signal.

Calibration runs each task many times per model and estimates pass rates or pass@k. Tasks that are never or always solved give outcome-based RL nothing to learn from, so they are revised, split or retired.

A model’s ability to invoke a function by emitting its name and structured arguments, then use the result.

Tools are described with JSON Schema so the model knows each argument’s type and constraints. The major model APIs each define a tool format, and MCP standardizes how tools are served to any client.

The full record of an episode: each observation, action and result, in order.

Trajectories are the raw material for supervised fine-tuning, imitation learning and debugging. Expert demonstrations recorded in the same environment as agent rollouts produce directly comparable trajectories. ATIF is one standard format for exchanging them.

Code that decides whether an agent’s episode succeeded by inspecting the final state and the action log.

A verifier returns pass or fail, usually with per-check details, and its output becomes the reward for training or the score for evaluation. Strong verifiers recompute correct answers from each seed’s starting state, check for collateral damage and are themselves tested with adversarial probes.

Systematic testing of a verifier before its results are trusted.

Typical probes replay the reference solution on several seeds (it must pass), submit an untouched episode (it must fail), run random agents (they must fail), drop the solution’s final step (it must fail) and add unrequested writes (they must fail). Sandworld publishes these results for every task, with an A–C grade, on each world’s page.