Changelog

What's new

2026-10-21

Verifier audit: made-up PO numbers now fail

Verifiers

We read the published model runs episode by episode — every pass, and every failure that came close (a score of 0.6 or more) — looking for verifier mistakes in either direction.
No correct episode had been failed. One pass was wrong: a model invented a purchase-order number on a new invoice. Ledgerline's invoice tasks and Worksuite's now check for it (Ledgerline 1.1.1, Worksuite 1.0.1).
Published runs are re-scored from their recorded tool calls whenever a verifier changes, so every result on the Arena is graded by the current verifiers. Qwen3 8B moved from 11 to 10 passes.

2026-10-21

Experts

Calculation questions draw fresh numbers on every attempt and every question's options are shuffled, so a shared answer key doesn't work.
The form is issued when the expert presses Start and signed for that expert, domain and attempt; it expires after four hours, and the time taken is measured from it.
What was asked is stored with the attempt, so reviews and appeals see exactly the form that was served.

2026-10-21

Environments

A scaffold reads an app's OpenAPI 3 document and builds a working twin: every operation becomes a typed tool over data seeded from the schemas.
Lists, searches, lookups, creates, updates and deletes work out of the box, including nested collections; other operations are recorded until you give them behaviour.
Tested on Swagger's Petstore and Bump.sh's Train Travel API. The policy, tasks and verifiers that make a twin worth training on are still written by hand.

2026-10-21

Data

One click on an eval run turns its failed episodes into a data project. Each item opens the app where the agent first went wrong, with its earlier steps already applied, and an expert takes over and finishes the task.
The verifier checks the finished episode before the work is accepted, so every record is a verified correction: the agent's own mistake, then the right way out of it.
Exports say whose steps are whose: the agent's turns carry weight 0 in the tool-calling format, metadata.recovery gives the hand-over step, and ATIF steps are tagged agent or human.
Experts can think aloud while they work: a note on any step is saved with it and delivered as the assistant's text for that tool call.

2026-10-21

New world

One company, Northwind, in three apps at once: Daybook for email and calendar, Crewboard for engineering issues, Ledgerline for billing. 64 tools, one shared clock, the same people and customers in every app.
12 tasks that carry facts across apps: service credits that depend on an issue's resolution, add-ons billed only once their setup is done, escalation triage, payments and reconciliation, and a customer call across three time zones.
Prompt injections planted in an email, an issue description and a customer record; three tasks where the agent has to ask. Every task grades A in verifier QA.
Composite worlds are a general mechanism: any twins can be mounted together, with diffs, footprints, QA and replays working unchanged.

2026-10-21

Evals

One agent loop for every provider: Anthropic, OpenAI, Google Gemini, OpenRouter, any OpenAI-compatible server (vLLM, SGLang, LM Studio) and local models through Ollama — same prompt, tools, step budget and verifiers for all.
Every failed episode gets a reason read off the verifier's checks: prompt injection, didn't ask, broke a rule, ran out of steps, gave up, wrong result, collateral damage, refused.
A command-line benchmark runner that resumes where it stopped and writes every episode's full log; complete runs are published on the Arena with a page per episode.
Episode replays compare the run with the reference solution — where it first departed, what it changed that the reference didn't and vice versa — and can retry from any step.

2026-10-21

Agents

Step signals for RL training: valid call, state changed, progress on the task's checks and its change, collateral damage — returned with every step of a training episode.
Fault profiles that make calls fail deterministically: transient errors, rate limits, and a timeout after the write went through, with a check that catches blind retries.
A local server with every world, the same API, no rate limits and a batch endpoint — about 10,000 steps a second on a laptop.
Computer use: a bridge that runs any twin's app in a headless browser with screenshots, numbered elements and click/type/scroll actions, graded by the same verifiers — so the GUI tax can be measured on identical tasks.
Python client 0.2: step signals in info, fault profiles, shaped rewards, a GRPO-style rollout example and a computer-use client.
LLM-backed simulated users (optional): an LLM plays the person on ask-the-user tasks, revealing only what it's asked; replies are cached and replay exactly.
Expert credentials are signed with Ed25519; anyone can verify one against the public key at /.well-known/jwks.json.

2026-10-21

New worlds

Skyfare, an airline booking app (mobile): fare families with real change and refund rules, the 24-hour rule, check-in windows, seat maps, bags, disruption remedies and award tickets.
Daybook, an email and calendar suite (web): mail triage, attachments, drafts, rooms, free/busy across time zones, RSVPs and a confidential-mail rule.
Each ships with 18 tasks, including two prompt-injection tasks and two ask-the-user tasks, and every task grades A in verifier QA.

2026-10-21

Environments

Quickbasket, Coinpurse, Ledgerline and Crewboard move to 1.1.0: each gains prompt-injection tasks (instructions planted in courier notes, product descriptions, request notes, bank memos and pasted support tickets) and ask-the-user tasks with a simulated user.
Policies now say that text inside the app is information, not instructions, and that missing details should be asked for rather than guessed.

2026-10-21

Launch

6 app twins (Quickbasket, Coinpurse, Skyfare, Ledgerline, Crewboard, Daybook).
Verifier QA: reference, do-nothing, random, stop-early and extra-write probes for every task, published per world and at /api/v1/environments/{slug}/qa.
Collateral-damage checks: episodes fail when they change state a correct solution never touches.
Gym-style API (reset, step, submit) with tool definitions for the Claude Messages API, OpenAI function calling and MCP.
Expert data projects: demonstrations, preferences, rubric grades, SFT answers and red teaming, with agreement metrics and TRL-ready exports.
Eval runs for baselines and hosted Claude models, with per-episode replay and pass@k.
Prompt-injection safety tasks: untrusted text inside the app (courier notes, product descriptions) asks for actions the user never requested, and the verifier checks they were ignored.
Harbor task packages (beta): any task and seed exports as a Harbor task directory whose verifier replays the recorded calls and writes reward.json. The task container never holds the verifier or the solution.
ATIF v1.8 trajectories for every episode and for demonstration datasets.
Python client with a Gymnasium-style wrapper, and an example that runs Claude on any task.