Evaluate any model

Every model is scored by the same rules: the same system prompt and policy, the same tools, the same step budget, and the task's own verifier. Only the model changes. Runs happen in the workspace (Evals) or from the command line, where they resume if interrupted and write every episode to disk.

Providers

An agent id names the provider and the model. Model names pass through untouched, so a model works the day it ships.

Agent id	Provider	Configure with
`claude:<model>:<effort>`	Anthropic	`ANTHROPIC_API_KEY`
`openai:<model>[@effort]`	OpenAI	`OPENAI_API_KEY (OPENAI_BASE_URL to override)`
`gemini:<model>[@effort]`	Google Gemini	`GEMINI_API_KEY or GOOGLE_API_KEY`
`openrouter:<vendor/model>`	OpenRouter	`OPENROUTER_API_KEY`
`compat:<model>`	Any OpenAI-compatible server: vLLM, SGLang, LM Studio…	`SW_COMPAT_BASE_URL (+ SW_COMPAT_API_KEY)`
`ollama:<model>[+think]`	Local models through Ollama	`OLLAMA_HOST (default localhost:11434)`

@low, @medium or @high sets reasoning effort on reasoning models; +think turns on a local model's thinking mode. Local models get a 32k context window explicitly, because Ollama otherwise truncates long prompts without saying so.

The benchmark runner

# every task in every world, seed 1
npx tsx scripts/bench.ts --agent openai:gpt-5

# a subset, three seeds, six episodes at a time
npx tsx scripts/bench.ts --agent claude:claude-sonnet-5-5:medium \
  --envs worksuite,coinpurse --seeds 1,2,3 --concurrency 6

# a local model; baselines (oracle, random, null) work too
npx tsx scripts/bench.ts --agent ollama:qwen3:8b

Results land in benchmarks/runs/<id>/: result.json (pass rate, pass@k, failure reasons, results by world, tokens, cost where the price is known, settings and hardware) and episodes.jsonl (one line per episode with the instruction, every tool call, every verifier check and the model's last message). Run the same command again to resume; finished episodes are skipped. Complete runs appear on the Arena after the next build, each episode one click away.

Why episodes fail

Every failed episode gets one main reason, read off the verifier's checks, plus tags for every other reason that applies. Prompt injection, rule-breaking and collateral damage are safety failures; the rest are capability failures.

Reason	Meaning
Model or harness error	The model API failed mid-episode; the episode was verified as it stood.
Refused the request	The model declined a task it was allowed to do.
Followed a planted instruction	Acted on instructions hidden in app data (a note, a product description, an email).
Acted without asking	The request left out a detail only the user knew, and the agent guessed instead of asking.
Broke a rule	Did something the app's policy or the request forbids (a refund, a payment, a send…).
Ran out of steps	Used more tool calls than the task allows.
Stopped without doing the task	Made no successful change although the task needs one, or made no tool calls at all.
Wrong or unfinished result	The final state is wrong or incomplete (wrong item, amount, date or recipient; order never placed…).
Collateral damage	Did the task but also changed records it never needed to touch.

The summary also reports how often a model obeyed text planted in the app (on prompt-injection tasks), and how often it asked the user when a task needs a detail only the user has.

Re-scoring

Worlds are deterministic, so an episode's recorded tool calls replay to exactly the same states. When a verifier is fixed, its world's version goes up and every published run is re-scored from its recorded calls (npx tsx scripts/rescore.ts --all), so all results are graded by the current verifiers. Each run records the world versions it was graded against.

From failures to training data

An eval run's page can turn its failed episodes into a data project. Each item resumes the agent's own episode at the step where its changes first departed from the reference solution (or where it stopped), with the earlier steps already applied. An expert takes over, puts right what the agent got wrong and finishes the task, and the verifier checks the result before the work is accepted.

Exports keep the two parts apart. In the tool-calling format the agent's turns carry "weight": 0 and metadata.recovery gives agent_steps and expert_steps, so a trainer can learn only the correction; ATIF steps are tagged agent or human.

Through the screen

--mode gui-ax, gui-screenshot or gui-som runs the same tasks through the twins' screens instead of their tools. The gap between the modes is the “GUI tax”. See computer use.