On station four of the collection floor, an operator sets down her leader arms after a clean bowl-to-sink transfer, and the batch dashboard flags the episode anyway. The instruction field reads “put the bowl in the sink,” the same sentence shape as two hundred episodes before it, and the template-share gate has just tripped. That flag, firing in the middle of a live program, is the single most important control in the engagement this post describes.
The client behind that program, a humanoid foundation model team, did not come to us asking for data. They came asking why their model was failing. Their OpenVLA-style fine-tune scored well on standard tabletop evaluations, then collapsed on the bimanual kitchen tasks their product actually needed. Worse, the policy only responded reliably to instructions phrased exactly like the training labels. “Put the bowl in the sink” worked. “Move that bowl into the sink” mostly did not.
The failure had a clean explanation in their data, and it is the most common one we see. Their existing corpus was large by episode count, roughly three times what we ended up collecting, but it covered only 9 task families and used 14 instruction templates. The model had learned those 9 tasks and those 14 sentence shapes, thoroughly, and almost nothing else. Volume was never the problem. Distribution was.
This post walks through what we scoped, what the QA gates caught, what it cost in structure (with our standard benchmark rates), and what changed in their evaluations. If you are planning a fine-tuning collection, the transferable part is the method: diversity targets set as hard gates, not aspirations.
Key Takeaways – 1,400 hours of bimanual teleoperation across 42 task families replaced a corpus 3x larger in episode count but 5x narrower in tasks. – A hard instruction diversity gate (no template above 8% of any batch) fixed paraphrase brittleness; language annotation ran at our standard $12-15 per data-hour, segment-level. – Blending new data 60/40 with the legacy corpus roughly doubled held-out kitchen task success in the client’s own evaluation harness. – QA rejected 11% of collected episodes; the top three causes were teleoperator mid-chunk pauses, frame-action sync drift, and failed-but-unlabeled episodes. – Total program ran 19 weeks from scoping to final delivery in LeRobot-compatible format.
The Starting Point: A Large but Narrow Corpus
A narrow corpus is a dataset whose episode count outruns its task, scene, and language diversity, and it trains models that look strong on in-distribution tests and brittle everywhere else. The client’s legacy data was a textbook case: high volume, 9 task families, 14 templates, single kitchen scene, one camera geometry.
The published evidence already pointed to the fix. Open X-Embodiment showed pooled diversity beating single-domain volume (arxiv.org/abs/2310.08864), and Mobile ALOHA reached strong per-task results co-training with roughly 50 demonstrations per task over a broad base (arxiv.org/abs/2401.02117). Our job was to apply that at commercial fine-tuning scale on their embodiment, where cross-embodiment transfer is weakest.
What We Scoped
The collection specification is the deliverable that matters most, so we fixed it before a single episode was recorded:
- Volume and spread: 1,400 robot-hours of bimanual teleoperation across 42 task families, capped at 60 hours per family to force breadth.
- Scenes and objects: 6 kitchen layouts, 300+ object instances, mandated lighting variation across sessions.
- Capture spec: 4 camera streams per station (two exocentric, head, wrist), 30 Hz, hardware-timestamped, frame-to-action skew held under 10 ms.
- Language spec: natural free-form instructions written at collection time, segment-level labels with timestamps on all long-horizon episodes, and the diversity gate below.
- Format: LeRobot-compatible episodes so their training stack ingested data on day one (github.com/huggingface/lerobot).
The Instruction Diversity Gate
An instruction diversity gate is a batch-level QA rule that rejects any 1,000-episode batch where a single instruction template exceeds a set share, in this case 8%. Annotators drift toward comfortable phrasings; without a gate, “pick up the X and place it in the Y” quietly becomes half your labels.
We enforced three thresholds per batch: template share under 8%, minimum unique verb count, and minimum spatial-relation phrase coverage (“behind the cutting board”, “second shelf from the top”). Batches that failed went back for paraphrase augmentation before delivery. This is also where the money went: segment-level annotation with paraphrase passes ran at our standard $12-15 per data-hour, versus $8-10 for a plain episode-level pass. The client initially pushed back on that line item. It ended up being the line item their evaluation results vindicated most directly.
How the Program Ran
A collection program’s timeline is set by station-hours and reset speed, not by calendar ambition, so we planned backwards from throughput. The program ran 19 weeks across six bimanual stations: two weeks of scoping and task-family design with the client’s ML team, one week of operator training and pilot batches, fourteen weeks of production collection, and two weeks of final QA, re-collection of rejected hours, and packaging.
The pilot week matters more than its size suggests. We shipped the first 40 hours to the client immediately and they ran a smoke-test fine-tune before full production started. That surfaced two spec problems while they were still cheap to fix: their training stack expected a different gripper action convention, and one camera’s mounting height put the left workspace edge out of frame on tall-object tasks. Catching either in week twelve instead of week three would have cost hundreds of re-collected hours. If a vendor does not offer a pilot-batch gate before full-rate production, build one into the contract yourself.
Weekly deliveries went out in LeRobot format with per-batch QA reports, so the client’s dataset grew continuously instead of arriving as one end-of-program surprise.
What QA Caught
Episode rejection is the health metric of a collection program, and ours ran at 11% across the engagement. Where those episodes died:
| Rejection Cause | Share of Rejects | Why It Matters for VLA Training |
|---|---|---|
| Teleoperator mid-chunk pauses | 34% | Corrupts action-chunk targets for chunked policies (ACT-style and continuous heads) |
| Frame-action sync drift > 10 ms | 27% | Measurably degrades continuous-action fine-tunes |
| Failed episode, unlabeled | 22% | Success mislabeling flattens gains on precision tasks |
| Dropped frames / camera faults | 11% | Breaks observation continuity |
| Instruction-episode mismatch | 6% | Directly poisons language conditioning |
Every rejected hour was recollected within the fixed budget because our per-hour rates ($40-60 for bimanual leader-follower work, our standard benchmark range) price in an expected rejection band. If a vendor quotes you a rate with no stated rejection policy, that risk is sitting on your side of the table.
Results
The client fine-tuned on the new corpus blended 60/40 with their legacy data and ran their internal evaluation harness before and after. Directional results, their harness, not an audited benchmark:
- Held-out kitchen task success roughly doubled against the legacy-data baseline.
- Paraphrased-instruction success moved from near-random to parity with template phrasing.
- Long-horizon task completion improved most on tasks with segment-level labels, consistent with what the pi-zero pipeline reports about post-training data quality (arxiv.org/abs/2410.24164).
The result we quote most often is the second one, because it isolates the language layer. Nothing about the model changed. The labels changed.
What Transfers to Your Program
Four rules from this engagement apply to almost any VLA fine-tuning collection:
- Audit distribution before buying volume; count task families and unique templates in your existing corpus first.
- Set diversity thresholds as batch-level rejection gates, not guidelines.
- Price annotation and QA as separate line items so you can see what you are cutting when you cut.
- Cap hours per task family; the marginal episode of a covered task is your most overpriced purchase.
The full scoping framework, cost tables, and the RFP scorecard we build these programs from are in the complete guide to VLA training data.
Next Step
if your evaluations look like this client’s before-picture, book a scoping call. We will audit your corpus distribution before recommending a single collection hour, or start with the complete VLA training data guide.
Frequently Asked Questions
How many hours of data does it take to fine-tune a VLA for a humanoid?
This program used 1,400 hours across 42 task families, blended with existing data. Humanoids sit at the expensive end because cross-embodiment transfer from open datasets is weakest for them; standard single arms often need only 200-500 hours.
What is an instruction diversity gate?
A batch-level QA rule that rejects annotation batches where any single instruction template exceeds a set share (we used 8%), with minimum thresholds for unique verbs and spatial-relation phrases. It prevents the label monoculture that makes VLA policies brittle to paraphrasing.
Why blend new data with a legacy corpus instead of replacing it?
The legacy data still carried valid task knowledge; the 60/40 blend kept that signal while the new data corrected the distribution. Full replacement wastes paid-for coverage and can regress previously working tasks.
What rejection rate is normal in teleoperation collection?
We ran 11% on this program, and 10-15% is the band we consider healthy for bimanual long-horizon work. A vendor reporting near-zero rejection is usually not checking sync, pauses, or success labels.
