Expert data
Human judgment where it matters. Checked like code where it can be.
What we produce
Five kinds of data, one quality system.
Agent demonstrations
Experts complete tasks inside the twins, from scratch or by taking over a failed agent episode where it went wrong (recovery data), with their reasoning for each step if you ask for it. Each recording is a tool-call trajectory the task verifier has already checked — failed recordings never ship.
- TRL tool-calling messages — {"messages", "tools"} with tool_calls and tool results, verified episodes only
- ATIF v1.8 trajectories — Harbor's trajectory interchange format, one trajectory per line
- Raw episodes — Instruction, full action log and verifier checks
Preference pairs
Pairwise judgments with confidence and a written deciding factor, double-annotated by default. Ties are excluded from chosen/rejected exports.
- TRL preference (DPO) — {"prompt", "chosen", "rejected"} from majority judgments
- All judgments — Every individual judgment with confidence and rationale
Rubric grades
Criterion-level scores from credentialed reviewers against your rubric, with a weighted total and written error notes.
- Rubric scores — Per-criterion scores, weighted score and reviewer notes
SFT answers
Ideal responses written — and run, for code — by practitioners, reviewed against your style and correctness bar.
- TRL conversational — {"messages": [user, assistant]} — SFTTrainer-ready
- Prompt / completion — {"prompt", "completion"}
Red teaming
Policy-breaking attempts against your agent with the exact transcript, observed behaviour, severity and reproducibility.
- Red-team findings — Goal, attack, observed behaviour and ratings
Every delivery also includes
- Dataset card — Hugging Face-style README with collection process, quality and provenance
- Provenance — One row per delivered record: item, pseudonymous annotator, review, timestamps
Quality
Numbers you can check, not adjectives.
Quality is reported per project and recomputed from the raw judgments every time you open it.
Gold tasks
Known-answer items are mixed into queues; each annotator carries a running gold accuracy.
Agreement
Krippendorff's alpha on double-annotated items — nominal for choices, interval for scores.
Verifier checks
Demonstrations are accepted only when the task verifier passes on the final app state.
Activity signals
Active time, focus changes and paste events flag low-effort or pasted work for review.
Written rejections
Every rejection carries a reason; experts can appeal and get an answer within 72 hours.
Provenance
Per-record project, item, pseudonymous annotator, timestamp and review verdict.
How a project runs
From rubric to dataset in four steps.
- 01
Scope
Agree the task, rubric, qualification bar, volume and turnaround. Price is per accepted task.
- 02
Calibrate
A pilot batch with your team reviewing alongside ours until agreement and gold accuracy hold.
- 03
Produce
Vetted experts work through the queue; disagreements go to adjudication, rejections to appeal.
- 04
Deliver
Download any time in your format, with the dataset card and provenance — or pull by API.
FAQ
Common questions
What formats do you deliver expert data in?
JSONL that loads straight into common trainers: TRL conversational messages (with tools and tool calls for trajectories), prompt/completion, and prompt/chosen/rejected for preference training — plus a dataset card and a provenance CSV with one row per record.
How do you measure label quality?
Every project reports acceptance rate, Krippendorff's alpha on double-annotated items, gold-task accuracy per annotator and median time on task. Demonstrations are additionally checked by the environment's verifier before acceptance.
Do buyers see who the annotators are?
No. Annotators appear under a stable per-project pseudonym, so you can group or filter by annotator without learning identities.
How do you detect low-effort or AI-generated work?
The workbench records activity signals — active time on task, focus changes, paste events and pasted character counts — never keystroke content or screen recordings. Combined with gold tasks and agreement checks, these flag work for review.
Need data that matches a twin?
Pair a custom environment with expert demonstrations recorded inside it — the verifier checks both.