A computer-use environment lets an AI agent operate software the way a person does — through screenshots, clicks, typing and scrolling — or through the app’s own tools, inside a resettable copy of a desktop, browser or phone. Mobile is the least-served surface: the best-known public Android benchmark has saturated, and almost no vendor sells deterministic phone-app replicas with state-level verifiers.
What is a computer-use agent?
A computer-use agent controls software through its interface instead of an API. At each step it observes the screen — as pixels, an accessibility tree or both — and emits an action. The major model providers each define their own action schema:
- OpenAI’s computer actions: click, double-click, drag, keypress, move, screenshot, scroll, type and wait.
- Anthropic’s computer-use toolset,
computer_toolset_20260801, with 17 tools. - Gemini’s computer use, with browser, desktop and mobile environments, coordinates normalized to 0–999, and mobile actions such as
open_app,list_appsandlong_press. - AndroidWorld’s JSONAction: click, double_tap, long_press, swipe, scroll, input_text, open_app, navigate_back, navigate_home, keyboard_enter, wait, status and answer.
What surfaces can an agent environment expose?
| Surface | Agent observes | Agent acts with | What it tests |
|---|---|---|---|
| Tools / MCP | Structured tool results | Typed function calls | Planning, tool choice, policy |
| Accessibility tree or DOM | Elements with stable IDs | BrowserGym-style click(bid), fill(bid, value) | Navigation, form filling |
| Pixels | Screenshots | Coordinates and keystrokes | Visual grounding, plus everything above |
| Mobile actions | Screenshots and a UI tree | Taps, swipes, long presses, app launches | Touch interaction, app switching |
Tool surfaces are the cheapest and most deterministic; pixel surfaces are closest to how people use software. The best environments offer more than one door into the same state, so you can tell perception failures from reasoning failures.
Why are mobile environments harder to build?
- Infrastructure. Android emulators need hardware virtualization (KVM) and budgets of 2–5 GB or more of RAM each (AndroidWorld, large-scale mobile RL).
- Real apps resist automation. Emulators fail Google’s Play Integrity checks, and WeChat, Alipay and Taobao reportedly blocked ByteDance’s phone agent, with 8 of 23 apps restricting it (36Kr, 36Kr follow-up).
- Real apps can’t be reset. Accounts keep history, servers change, and actions have consequences — an agent can actually buy the groceries.
- iOS is constrained. Apple’s macOS license limits virtualization to two extra VMs per Mac, and sets a 24-hour minimum for leasing a Mac to someone else.
Is AndroidWorld saturated?
Yes. AndroidWorld — 116 tasks across 20 Android apps — was hard when it launched. Its leaderboard now shows five entries at 100% against a human baseline of 80%, and Minitap claims 116 of 116. Successor benchmarks have followed the same arc, climbing from roughly 50% to 80–98% within four to eight months (MobileWorld, MobileRun).
The lesson isn’t that mobile agents are solved; it’s that fixed public task sets get solved. What’s scarce are fresh, private, harder suites: multi-app tasks (62% of MobileWorld’s tasks span more than one app, per its paper), long horizons, policy constraints and UI variation — OpenApps found success rates swinging by more than 50% across app variants.
Why use app twins instead of real apps?
A twin is a working replica of an app category with its own backend: the same kinds of screens, data and rules, under your control. MobileWorld’s authors took this route, building replica apps — including a React Native Gmail clone, Mattermost and a mock e-commerce app — restoring emulator snapshots and backend databases for every episode, and grading without an LLM judge.
Twins give you what real apps can’t: an exact reset, seeded variants, safe side effects and a verifier that can read the database. The trade-off is fidelity — a twin is only as realistic as its builders make it — so it’s worth validating some results on real devices. Tooling exists for that: Scale’s AgentEnv plugin for MobileRun drives real cloud iPhones, though apps keep saved state between runs and the grade is a single check of final-screen text.
How should a mobile task be verified?
By reading state, not pixels. A screenshot shows what the app displayed; the database shows what actually happened. AndroidWorld’s task checks read app databases over ADB, and MobileWorld grades from backend state without an LLM judge. A grocery order placed on a phone should be verified exactly like one placed on the web: one new order, the right items, the right slot, nothing else changed.
Screen checks still have a place for tasks whose whole point is what the user sees, such as a confirmation being shown. Treat them as a supplement, and never let a final-screen text match stand in for a state check.
Who wants mobile agents?
The demand signals are real but uneven. Android is shipping agentic UI automation for food-delivery, grocery and rideshare apps, and Gemini’s computer-use API has a mobile environment. Chinese labs run between 256 and about 10,000 parallel mobile environments in-house (one report, MAI-UI). US labs’ public focus is browsers and desktops — OpenAI’s computer-use hiring targets “browsers and desktops”. Mobile demand is emerging rather than proven, and anyone selling mobile environments, including us, should say so.
How do Sandworld’s mobile twins work today?
Quickbasket (grocery delivery) and Coinpurse (a payments wallet) are mobile-style twins: they’re designed as phone apps, and their tasks are the kind people do on phones. Today they run in the browser, not on Android or iOS. Each twin has two surfaces over one state:
- A tool surface for agents: typed tools served in Claude Messages API, OpenAI function-calling and MCP formats.
- A phone-frame UI for people. Every tap issues the same tool call an agent would make, so a human demonstration and an agent rollout produce identical action logs.
That design keeps episodes cheap, fast and exactly reproducible, and it makes expert demonstrations directly comparable with agent runs. It doesn’t test pixel-level perception or real touch input: agents currently act through tools only. If your target is visual grounding on native Android, our twins test the planning and policy half of the problem, not the perception half.
What should a good mobile agent environment include?
- A per-episode reset of the device snapshot and the backend database, with a frozen clock
- Seeded parameter variants, with verifiers that recompute answers
- State-level checks, including collateral damage, and no LLM judge where state suffices
- More than one action surface: tools, a UI tree with stable element IDs, and pixels
- Adapters for common action schemas such as AndroidWorld’s and Gemini’s
- Multi-app and long-horizon tasks, plus UI variants
- Published per-task resource requirements and verifier QA results
For the fundamentals behind this checklist, see what an RL environment is and how environments differ from static benchmarks.