Skip to content
Sandworld

Computer use

Every twin has a visual app, and every tap in it becomes the same tool call an API agent makes. The computer-use bridge opens a twin in a headless browser and lets an agent operate it like a person: look at the screen, click, type, scroll. The episode is graded by the task's own verifier, exactly as for tool use, so the same model can be compared across interfaces on the same tasks.

Start the bridge

npx tsx scripts/cua-bridge.ts --site http://localhost:3000   # → http://127.0.0.1:8790

The bridge opens /play/<world>?task=…&seed=…: the twin alone, at 390×844 for mobile twins and 1280×800 for web twins, with no page chrome or submit button for a model to click.

Observations

  • Element list — every visible, unobstructed control with a number (Set-of-Marks), role, accessible name, value and box.
  • Visible text — what's on screen, for text-only models.
  • Screenshot — a PNG, and optionally a copy with each element's number drawn on it.

Actions

POST /sessions              { "env": "quickbasket", "task_id": "qb-01", "seed": 1, "marks": true }
POST /sessions/{id}/act     { "action": "type", "element": 3, "text": "milk", "press_enter": true }
                            { "action": "click", "element": 7 }
                            { "action": "select", "element": 4, "option": "Net 30" }
                            { "action": "key", "key": "Escape" }
                            { "action": "scroll", "direction": "down" }
                            { "action": "ask_user", "message": "Which address?" }
POST /sessions/{id}/submit  → the verifier's verdict

From Python

from sandworld import ComputerUse

s = ComputerUse().start("quickbasket", "qb-01", seed=1, marks=True)
print(s.instruction)
search = next(e for e in s.observation["elements"] if e["name"] == "Search products")
s.type(search["id"], "milk", press_enter=True)
verdict = s.submit()
print(verdict.passed, verdict.failed_checks)

Measure the GUI tax

The benchmark runner drives any model through the screen in three modes — gui-ax (element list and text), gui-screenshot and gui-som (marked screenshot) — with the same prompt, policy and verifiers as tool use. Older screens collapse in the conversation so long episodes stay within context.

npx tsx scripts/bench.ts --agent claude:claude-sonnet-5-5:medium --mode gui-som \
  --site http://localhost:3000 --envs quickbasket,ledgerline

scripts/cua-check.ts solves a task through the GUI alone, as a reference that the path works end to end. Results appear on the Arena labelled by interface.