Beta

Harbor task packages

Labs run evaluations and RL in their own harnesses. Every Sandworld task can be exported as a directory in the Harbor task layout, so a twin plugs into an existing pipeline without our SDK. Each package is one task at one seed; export several seeds to get distinct starting states for the same request.

What's in a package

Path	Contents
`instruction.md`	The user's request, how to act with the sw command, and the app's policy. The task title is left out because titles like “Refuse a claim…” give the answer away.
`task.toml`	Harbor task configuration (schema 1.3): name, keywords, metadata (environment, version, seed, difficulty, skills, verifier QA grade, canary), timeouts, a separate verifier sandbox with no network, and an MCP server entry.
`environment/`	A Dockerfile on node:20-bookworm-slim plus the twin runtime: the sw CLI, the MCP server, the app's tools and this task's starting state.
`solution/solve.sh`	The reference solution as sw call commands, generated from the environment's own reference solver.
`tests/test.sh`	The verifier. Rebuilds the seeded start, replays the recorded calls and writes reward.json, reward.txt and checks.json to /logs/verifier.

Why the answer can't leak to the agent

The task container holds the twin's tools and the starting state only. The verifier and the reference solution are in tests/ and solution/, which the harness uploads after the agent stops, so an agent with a shell can't read them.
Every sw call is appended to /sandworld/episode.json, which task.toml lists as an artifact so it reaches the verifier sandbox. The verifier trusts only the list of calls: it rebuilds the seeded start itself and replays each call through the same tools. A log edited by hand can't claim a state the tools didn't produce, and a log for another task or a corrupt file scores 0.
Scoring includes the task's checks plus the generic ones: no collateral changes outside what a correct solution touches, and the step budget.
An internal verifier error exits non-zero without writing a reward, so the harness records an infrastructure fault rather than a score of 0.
Every file carries the environment's canary GUID, so task data that ends up in a training corpus can be found.

Get packages

From a checkout of the platform, for any environment, tasks and seeds:

npm run export:harbor -- --env quickbasket --seeds 1,2,3
npm run export:harbor -- --env coinpurse --tasks cp-01,cp-02 --tar
# → exports/harbor/sandworld-<env>-<task>-s<seed>/ and manifest.json

Or one package at a time from the API. Packages include the verifier, so this endpoint needs a workspace API key with the environments scope (part of an environment licence); the sandbox API stays open.

curl -sf -H "authorization: Bearer $SANDWORLD_KEY" \
  -o qb-01-s1.tar.gz \
  "https://sandworld.dev/api/v1/environments/quickbasket/tasks/qb-01/harbor?seed=1"
tar -xzf qb-01-s1.tar.gz

Inside the task container

sw task                                      # the request, policy and budget
sw tools                                     # tools and parameters (--json for JSON Schemas)
sw call get_order '{"order_id": "QB-10388"}'
sw history                                   # calls made so far
sw mcp                                       # the same tools over MCP (stdio)

Tool results are JSON on stdout; a failed call exits with status 1 and prints the error and its code. Calls are final, as in the real app. MCP clients see each tool with readOnlyHint and destructiveHint annotations.

Rewards

/logs/verifier/reward.json holds labelled numeric metrics:

Key	Meaning
`reward`	1 if every weighted check passed, else 0
`score`	Weighted share of checks passed (partial credit), 0–1
`checks_passed / checks_total`	Counts of weighted checks
`steps / write_steps / errors`	Tool calls made, state-changing calls that succeeded, and calls that failed

reward.txt repeats the binary reward for harnesses that read only that file, and checks.json lists every check with its detail.

Try one without a harness

cd sandworld-quickbasket-qb-01-s1
docker build -t sw-task environment
docker run --rm -it -v "$PWD/solution:/solution" -v "$PWD/tests:/tests" sw-task bash
bash /solution/solve.sh && bash /tests/test.sh && cat /logs/verifier/reward.json

Beta limits

We test packages by running solve.sh and test.sh outside Docker in CI. Validate them with your Harbor version before a large run, and tell us what breaks.
The starting state is a file inside the container. An agent could read it directly instead of using the tools. It still can't change the outcome without making tool calls, but for strict evaluations use the hosted API or MCP server, where state never leaves our side.
Mobile twins are packaged with their tool surface. Android emulator images for screen-based agents are on the roadmap.