To buy expert training data well, specify three things before any work starts: the exact format your training code ingests, the expertise each item requires, and the quality evidence you’ll accept — gold-task accuracy, agreement between independent reviewers, reviews per item and per-row provenance. Then run a small pilot, audit it yourself, and pay for accepted items rather than hours.
Demand is shifting from volume to specialists. xAI reportedly cut about 500 generalist annotators while planning to scale specialists tenfold, and OpenAI’s GDPval accepted fewer than 10% of expert applicants, who averaged 14 years of experience. Here’s how to buy that kind of work without paying for noise.
What kinds of expert data can you buy?
| Type | What you receive | Typical use |
|---|---|---|
| Demonstrations | Recordings of an expert doing a task, with every tool call and result | Supervised warm starts, golden trajectories for evals |
| SFT answers | Ideal responses to prompts, written or checked by practitioners | Supervised fine-tuning |
| Preference pairs | Two responses with a judgment, a confidence level and a rationale | Reward models, DPO |
| Rubric grades | Criterion-by-criterion scores against a written rubric | Evaluation, rubric-based rewards |
| Red-teaming | Attempts to make a model break policy, rated for severity and reproducibility | Safety training and evaluation |
| Transcript corrections | Agent runs fixed step by step by an expert | Agent training — Anthropic’s hiring asks for exactly this |
Which file formats should you ask for?
Name them in the statement of work. If you train with Hugging Face TRL, its documented dataset formats cover most needs:
- Conversational SFT: a
messageslist of role and content turns, with an optionaltoolslist for tool-calling data. - Prompt–completion:
promptandcompletionfields. - Preference:
prompt,chosenandrejected. - Unpaired preference (KTO):
prompt,completionand a booleanlabel.
A conversational preference row looks like this:
{"prompt": [{"role": "user", "content": "I have $10,000 at 4.5% APY. About how much interest will I earn in a year?"}],
"chosen": [{"role": "assistant", "content": "About $450 before tax, if the rate holds and you don't add or withdraw money."}],
"rejected": [{"role": "assistant", "content": "About $4,500, since 4.5% of $10,000 is $4,500."}]}Other targets have their own shapes. OpenAI’s DPO format uses input, preferred_output and non_preferred_output, but OpenAI’s deprecation notices say its fine-tuning platform stops accepting new jobs from existing customers on 6 January 2027 and Evals shuts down on 30 November 2026. Amazon Bedrock’s Claude fine-tuning takes JSONL with system and messages. For agent trajectories, ATIF v1.8 records steps, tool calls and observations. Don’t anchor on one provider’s format.
What is data provenance, and why does it matter?
Provenance is the record of where each row came from: who made it, with what qualifications, under which guideline version, when, with what review and under what license. You need it to debug a dataset, to defend it legally, and to pull rows if a contributor’s work turns out to be bad.
It’s often missing: a large audit of public datasets found that hosting sites omit license information more than 70% of the time. Ask for:
- Per-row fields: a pseudonymous contributor ID, the verified credential, the guideline version, created and reviewed timestamps, the review chain and verdicts, and the license.
- Dataset documentation: a Hugging Face dataset card, Croissant metadata with its Responsible AI fields — annotation protocol, annotations per item, annotator demographics — and a datasheet.
- Evidence about AI use. OpenAI reportedly fired contractors caught using AI to grade, and Anthropic’s hiring calls for systems that “detect low-effort or adversarial work”. Time on task and paste events are useful signals.
How do gold tasks work?
Gold tasks are items with a known correct answer, mixed invisibly into an expert’s queue. Accuracy on gold measures each person continuously, not just at the entrance exam. A typical setup:
- Seed a fixed share of gold — say one item in ten — and refresh it so it can’t be memorized.
- Write each gold answer with an explanation, and review it like any other item.
- Act on trends rather than single misses: warn, retrain, then remove.
- Report gold accuracy per expert and per project with every delivery.
For demonstrations recorded inside an environment, the task verifier is a gold check on every item. A Quickbasket demonstration of qb-09 either files exactly one refund for two missing avocados and credits $2.98, or it doesn’t.
How should a vendor prove quality beyond gold?
Gold measures accuracy where an answer key exists. For judgment work, measure agreement: have two experts label the same items independently and report a chance-corrected statistic such as Cohen’s κ or Krippendorff’s α. For reference, GDPval reported 71% agreement between human graders, with about five reviews per task. Our guide to measuring annotator agreement walks through the math. Also ask for reviews per item, the rule for resolving disagreements, and the share of items sent back for revision.
What should a pilot look like?
- Write guidelines and acceptance criteria first, with worked examples of good and bad items.
- Start small — a few hundred items or fewer — with double review on all of them.
- Audit a random sample yourself before accepting the batch.
- Measure what the data does. The strongest evidence is lift on a model: Toloka reports that one epoch of its off-the-shelf RL data took Qwen3.5-27B from 41.6% to 64.5%, and xAI’s hiring tracks “data yield, eval lift”.
- Time it. Turnaround is part of quality; Anthropic’s hiring describes wanting an idea “on Monday… in the training mix by Friday”.
What belongs in the contract?
- An MSA and statement of work with acceptance criteria, rate cards, change control and audit rights — the terms OpenAI’s procurement hiring describes.
- Exclusivity, priced explicitly: Epoch AI reports that exclusive deals cost four to five times non-exclusive ones.
- Written IP assignment from every contributor, valid where each contributor lives.
- Security evidence. Scaled vendors such as Scale, Toloka and SuperAnnotate hold SOC 2 Type II and ISO 27001, and OpenAI reviews workforce vendors against ISO 27001, NIST 800-53 and SOC 2.
- Contributor identity checks. A July 2026 multi-government alert on North Korean IT workers tells platforms to review IDs strictly and interview in person.
Where does Sandworld fit?
Early, and we’d rather say so. Sandworld is a new company with no certifications yet — SOC 2 readiness is planned — and our expert program is just starting. What we offer is structural. Demonstrations are recorded inside our deterministic twins, so the task’s verifier checks every one before a person reviews it. Projects run with gold items and single, double or three-way review. Exports are TRL-ready JSONL — conversational messages, prompt–completion, prompt/chosen/rejected built from majority judgments (ties are held back), and tool-calling messages from verified episodes only — plus a Hugging Face–style dataset card and a provenance file with pseudonymous annotator IDs.
If you’re scoping a pilot, tell us what you need and we’ll say plainly whether we can do it.