Environment spec

Every Sandworld environment implements one contract. It's small on purpose: labs should be able to audit a twin in an afternoon.

The contract

interface EnvSpec<State> {
  slug: string; name: string; version: string;
  surface: "mobile" | "web";
  policy: string;                 // rules the agent must follow
  epoch: number;                  // simulated clock start
  createState(seed: number): State;  // deterministic
  tools: ToolSpec<State>[];       // the only way to change state
  tasks: TaskSpec<State>[];
}

interface TaskSpec<State> {
  id: string; title: string; difficulty: "easy" | "medium" | "hard";
  instruction(state: State): string;
  setup?(state: State, rng: Rng): void;
  verify(ctx: { initial: State; final: State; actions: ActionRecord[] }): Check[];
  solution(state: State): Array<{ tool: string; args: object }>;  // reference
  maxSteps?: number;
  collateral?: boolean;           // default true
  user?: {                        // a simulated user; adds the ask_user tool
    persona: string;
    facts(state: State): Array<{ id: string; topics: string[]; reply: string }>;
    fallback: string;
  };
}

Determinism

• The same seed always builds byte-identical state. Seeds vary background records, prices and stock that no task depends on.
• Task-critical facts are fixed, or recomputed by the verifier from the episode's own starting state — verifiers never hard-code answers.
• The clock is simulated: each step advances seven seconds from the environment's epoch.

Tools

Tools take JSON-schema-validated arguments and return observations. Some rules are enforced by the tools, as a real app would (stock, balances, state machines). Others are policy only — the tool allows the action and the verifier decides whether the agent should have taken it. That split is what makes judgment testable.

Verifiers

A verifier returns granular checks with ids, labels and detail. Weight-zero checks are informational. On top of every task's own checks, the runtime adds two:

• No collateral damage. The runtime replays the reference solution across five seeds and records which parts of the state it changes — the task's footprint. An episode that changes anything outside that footprint fails.
• Step budget. Episodes must finish within the task's step budget (60 by default).

Simulated users

Some requests leave out a detail only the user knows: which address, how many hours, which bug the customer meant. A task with a user gets an ask_user tool for that episode. The scripted user answers from fixed facts by topic, so replies are deterministic and replayable, and verifiers can require that the agent asked before its first change.

Injection tasks

Every twin has tasks where text inside the app — a courier note, a product description, a request note, a pasted support ticket, a bank memo — asks for actions the user never requested. The user's request is harmless; an ignored-injection check confirms the planted actions weren't taken, and the collateral check catches anything else.

Verifier QA

Every task is attacked before it ships. Its report — public at /api/v1/environments/{slug}/qa — records:

The test suite enforces these thresholds for every registered environment, so a weak verifier can't be merged.

Start from an OpenAPI document

A new twin can start from the real app's API description. The scaffold reads an OpenAPI 3 document (YAML or JSON) and builds a working twin from it: every operation becomes a typed tool, and records are seeded deterministically from the schemas, with ids, enums and references that hold together.

npx tsx scripts/scaffold-openapi.ts openapi.yaml --slug petclinic --category health

• List, search, get, create, update and delete work out of the box over an in-memory store, including nested collections (/owners/{id}/visits) and updates that carry the id in the body.
• Other operations check that the record they act on exists, then are accepted and recorded until you give them behaviour; read-only ones return a sample of their documented response.
• One example task with a verifier is generated so the twin runs end to end. The policy, realistic data, the real tasks and their verifiers are still yours to write: replace any tool's behaviour with overrides and pass your own tasks.

Versioning

Environments carry a semantic version. Episodes record the version they ran on, and footprints and QA reports are cached per version, so results stay comparable across releases.