To evaluate an AI agent, run it on realistic multi-step tasks in a resettable environment, grade the final state and action log with programmatic verifiers, and report pass rates across several seeds and repeated attempts, with error bars. Grade outcomes, not transcripts: did the right order get placed, and did nothing else change?
Most agent evals go wrong in one of three ways: tasks too vague to grade, graders that can be fooled, or samples too small to separate two agents. The method below avoids all three.
What should an agent evaluation measure?
A single pass rate hides most of what you need to know. Track these separately:
| Metric | Question it answers | How to measure it |
|---|---|---|
| Task success | Did the agent achieve the user’s goal? | Verifier checks on the final state; all must pass |
| Policy compliance | Did it follow the rules, including saying no? | Trap tasks where the forbidden action is possible |
| Collateral damage | Did it change anything it shouldn’t have? | Diff the final state against what a correct solution touches |
| Reliability | Does it succeed every time, or just once? | pass^k over repeated attempts |
| Efficiency | How many steps, errors and tokens did it need? | Action-log counts against a step budget |
| Partial progress | How close did failures get? | Share of checks passed, read per check |
Sandworld’s verifier reports most of these for every episode: a verdict, a score (the share of checks passed), each check with its detail, and counts of steps, state-changing steps and errored calls.
How do you build a good agent evaluation set?
- Start from real workflows. List what users actually ask an agent to do — reorder last week’s groceries, dispute a double charge — not what’s easy to grade.
- Write success as facts about state. “Exactly one new order, containing exactly these items, in this slot” is gradable. “Handled the request well” is not.
- Include tasks where the right move is to do less. In Quickbasket’s qb-05 the user asks to cancel “whichever of my orders hasn’t left the store yet”; another order is already out for delivery and must be left alone. In qb-10 the right answer is to refuse a refund.
- Add disambiguation. Coinpurse’s cp-02 asks the agent to pay Sam Rivera $45; the contacts also include Sam Rivers.
- Let seeds change the answers. In qb-02, “the cheapest 1 kg rice that is in stock” is Golden Grain at $3.29 on seed 1 but Harvest Brown at $3.69 on seed 7. The jasmine rice, often the cheapest on the shelf, is out of stock on every seed.
- Keep a held-out set. Tasks used for training or prompt tuning stop measuring generalization.
Why grade the final state instead of the transcript?
A transcript tells you what the agent said it did; the state tells you what it did. In qb-08 the user lists four items to buy “that are in stock… no substitutes”, and the spinach is out of stock. An agent that adds broccoli instead will still write a cheerful summary. The verifier’s item check fails with the detail p-banana×6, p-broccoli×1, p-coffee×1, p-yogurt-greek×2 — an exact record of the mistake.
Programmatic graders aren’t automatically right either. A 2026 audit of five public benchmarks found that 15.3% of FAIL verdicts were wrong. Audit graders the way you’d audit a model: run the reference solution, an agent that does nothing and random agents, then read a sample of verdicts by hand. Free-form outputs, such as an explanation to a customer, still need a rubric or a judge model whose agreement with human graders you’ve measured.
How many runs does an agent evaluation need?
More than most teams use. A pass rate from n episodes has a standard error of about √(p(1−p)/n). Suppose an agent passes 26 of 39 episodes — 13 tasks at three seeds each. That’s 66.7%, with a 95% interval of roughly ±15 points: anywhere from about 52% to 81%. A rival at 72% on the same set is not measurably better.
- Pair your comparisons. With a deterministic environment, run both agents on identical task–seed pairs and compare pair by pair. Paired differences are far less noisy than two independent pass rates.
- Repeat attempts. Models are stochastic even when environments aren’t, so run each task–seed pair several times.
- Pin the setup. Hold the environment version, tool definitions, step budget and hardware constant. Anthropic found that resource configuration alone moved Terminal-Bench 2.0 scores by 6 points.
- Separate infrastructure failures. A crashed sandbox says nothing about the agent; τ2-bench counts such errors separately instead of scoring them as failures.
What’s the difference between pass@k and pass^k?
They answer opposite questions. pass@k is the probability that at least one of k attempts succeeds — the right metric when a person or a test suite can pick the winner. pass^k is the probability that all k attempts succeed — the right metric when the agent acts unsupervised and every run counts.
For an agent that solves a task 70% of the time, pass@3 = 1 − 0.3³ = 97.3%, while pass^3 = 0.7³ = 34.3%. Same agent, same task. If your agent will issue refunds on its own, 34% is the number to worry about.
Which baselines should every agent eval include?
Three baselines need no model at all, and each one catches a different kind of broken task:
- Reference solver. Replays each task’s known-good solution. It must pass 100% of the time; if it doesn’t, the task or the verifier is broken.
- Do-nothing agent. Submits immediately. It must fail every task; if it passes one, a verifier is checking the wrong thing.
- Random agent. Calls random tools with plausible arguments. It sets the floor any real agent must clear.
Every Sandworld world publishes all three, plus the truncated and noisy probes described in our verifier guide. See them in Quickbasket’s verifier QA.
How do you turn eval results into decisions?
Read failures by check, not by task. If most Quickbasket failures land on slot and coupon checks, the agent struggles with constraints; if they land on “Other orders were left alone”, it over-acts. Each pattern points to a different fix: prompt changes, clearer tool descriptions, demonstrations or preference data. Re-run the same seeds after every change so you know an improvement is real.
What belongs on an agent evaluation checklist?
- Tasks phrased as real user requests, with success defined as facts about state
- Trap tasks where the right action is to refuse or do less
- Seeds that change the facts, and verifiers that recompute the answers
- Reference, do-nothing and random baselines
- Repeated attempts, reported as both pass@k and pass^k
- Paired comparisons on identical task–seed pairs
- A pinned environment version, tool set, step budget and resource profile
- Infrastructure failures counted separately
- A hand-audited sample of verdicts