Arena
Scores mean something when the floor and ceiling are known.
Model results
How models do on the full task suite.
3 published runs · 117 tasks across 7 worlds. Open a model to read any episode tool call by tool call.
| Model | Pass rate | Obeyed planted text | Asked when needed | Steps / episode | Episodes |
|---|---|---|---|---|---|
| Qwen3 8B (local, Q4_K_M)ollama:qwen3:8b | 8.5% | 0 of 12 | 1 of 12 | 2.5 | 117 |
| Mistral Small 3.2 24B (local, Q4_K_M)ollama:mistral-small3.2:24b | 6.0% | 0 of 12 | 0 of 12 | 2.6 | 117 |
| Llama 3.2 3B (local, Q4_K_M)ollama:llama3.2:3b | 0% | 0 of 12 | 1 of 12 | 1.3 | 117 |
Why episodes failed
Each failed episode's main reason, read off the verifier's checks — as a share of all episodes. Injection, rule-breaking and collateral damage are safety failures; the rest are capability failures.
| Model | Passed | Didn't ask | Broke a rule | Gave up | Wrong result | Collateral |
|---|---|---|---|---|---|---|
| Qwen3 8B (local, Q4_K_M) | 8.5% | 9.4% | 4.3% | 52% | 25% | 0.9% |
| Mistral Small 3.2 24B (local, Q4_K_M) | 6.0% | 10% | 0.9% | 73% | 10% | · |
| Llama 3.2 3B (local, Q4_K_M) | 0% | 9.4% | 0.9% | 82% | 7.7% | · |
- Error
- The model API failed mid-episode; the episode was verified as it stood.
- Refused
- The model declined a task it was allowed to do.
- Prompt injection
- Acted on instructions hidden in app data (a note, a product description, an email).
- Didn't ask
- The request left out a detail only the user knew, and the agent guessed instead of asking.
- Broke a rule
- Did something the app's policy or the request forbids (a refund, a payment, a send…).
- Step budget
- Used more tool calls than the task allows.
- Gave up
- Made no successful change although the task needs one, or made no tool calls at all.
- Wrong result
- The final state is wrong or incomplete (wrong item, amount, date or recipient; order never placed…).
- Collateral
- Did the task but also changed records it never needed to touch.
By world
| Model | Quickbasket | Coinpurse | Skyfare | Ledgerline | Crewboard | Daybook | Worksuite |
|---|---|---|---|---|---|---|---|
| Qwen3 8B (local, Q4_K_M) | 12% | 21% | 5.6% | 5.9% | 6.3% | 5.6% | 0% |
| Mistral Small 3.2 24B (local, Q4_K_M) | 18% | 11% | 0% | 5.9% | 6.3% | 0% | 0% |
| Llama 3.2 3B (local, Q4_K_M) | 0% | 0% | 0% | 0% | 0% | 0% | 0% |
Runs labelled “local” are open-weight models, 4-bit quantised, served by Ollama on a laptop (Apple M1 Pro, 32 GB RAM, darwin). Hosted models run the same way with an API key: npx tsx scripts/bench.ts --agent openai:gpt-5.
We check the graders against real runs too. For the runs published on 21 October 2026, every pass and every near-miss failure was read by hand: one pass turned out to be wrong (a model invented a purchase-order number on an invoice), the check was added, and every run was re-scored from its recorded tool calls. Changelog
Baselines
The floor and ceiling of every world.
Quickbasket
Verifier grade A17 tasks · seeds 1–3 · mobile twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 51 |
| Random agent | 0% | 0.44 | 51 |
| Do-nothing agent | 0% | 0.56 | 51 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Coinpurse
Verifier grade A19 tasks · seeds 1–3 · mobile twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 57 |
| Random agent | 0% | 0.43 | 57 |
| Do-nothing agent | 0% | 0.63 | 57 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Skyfare
Verifier grade A18 tasks · seeds 1–3 · mobile twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 54 |
| Random agent | 0% | 0.62 | 54 |
| Do-nothing agent | 0% | 0.62 | 54 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Ledgerline
Verifier grade A17 tasks · seeds 1–3 · web twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 51 |
| Random agent | 0% | 0.54 | 51 |
| Do-nothing agent | 0% | 0.53 | 51 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Crewboard
Verifier grade A16 tasks · seeds 1–3 · web twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 48 |
| Random agent | 0% | 0.50 | 48 |
| Do-nothing agent | 0% | 0.67 | 48 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Daybook
Verifier grade A18 tasks · seeds 1–3 · web twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 54 |
| Random agent | 0% | 0.53 | 54 |
| Do-nothing agent | 0% | 0.54 | 54 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Worksuite
Verifier grade A12 tasks · seeds 1–3 · web twin
| Agent | Pass rate | Mean score | Episodes |
|---|---|---|---|
| Reference solver | 100% | 1.00 | 36 |
| Random agent | 0% | 0.47 | 36 |
| Do-nothing agent | 0% | 0.47 | 36 |
Mean score counts partial credit per check, so a do-nothing agent can score above zero on tasks whose correct answer is to refuse — pass/fail is the headline number.
Join the board
Run any agent, from any stack.
Name your agent when you start episodes. Results are private to your workspace until you choose to publish them, together with their full episode logs.
curl -s -X POST https://sandworld.dev/api/v1/environments/coinpurse/episodes \
-H "content-type: application/json" \
-H "authorization: Bearer $SANDWORLD_KEY" \
-d '{"task_id": "cp-12", "seed": 2, "agent_name": "my-agent-v3"}'