Somewhere on every VLA data plan sits a line that reads “annotation and QA,” and it is the checklist item nearly everyone skips when the budget gets real. Collection hours get scoped to the dollar. GPU spend gets scoped to the node. The layers that decide whether those hours train anything get a checkbox. That skipped line is usually where we find the explanation when a team arrives with a Prismatic-style backbone, sensible action head, clean training code, and a model that still cannot pick up a mug it has never seen when the instruction is phrased slightly differently.
The deeper reason is structural, not a bug. Language models got to scrape a web that humanity spent thirty years writing. VLA models need trajectories where a physical robot moved and every action was recorded, and that data did not exist until someone paid to create it, one teleoperated episode at a time. Open X-Embodiment, the largest pooled open corpus, holds just over one million trajectories from 22 embodiments (arxiv.org/abs/2310.08864). Web-scale image-text corpora hold billions of pairs. Physical AI is trying to reach foundation-model behavior on roughly one-thousandth of the data its sibling fields consider table stakes.
This post argues one thesis: the VLA data bottleneck is not a single constraint but a four-layer stack (collection throughput, action-label fidelity, language annotation, QA), and teams that budget only for the first layer buy datasets that train worse models than their size suggests. We show where each layer binds, what it costs to push against, and which strategies the leading labs use to spend around it efficiently. We collect and QA this data for a living at DexSet, so the cost numbers are our own production benchmarks, flagged as such.
Key Takeaways – The robot data gap is around three orders of magnitude versus web vision-language corpora, and it cannot be closed by scraping. – The bottleneck has four layers: collection throughput, action-label fidelity, language annotation quality, and QA. Most teams only budget for the first. – RT-1 took 17 months and 13 robots to produce ~130k episodes; that throughput ceiling is why teleoperation economics matter. – In our benchmarks, usable teleop data costs $28-60 per robot-hour to collect, plus $8-15 per hour for language annotation and $5-9 for QA. – The efficient escape routes are cross-embodiment pretraining, web co-training, and spending fresh budget on diversity rather than raw episode count.
What Makes VLA Training Data a Bottleneck?
The VLA data bottleneck is the mismatch between how fast robot demonstration data can be produced and how much of it foundation-model training consumes. Every VLA needs episodes containing synchronized observations, executed actions, and language instructions, and each of those three elements has its own production constraint.
Consider the throughput math. RT-1’s dataset took a fleet of 13 robots 17 months to reach roughly 130k episodes across about 700 tasks (arxiv.org/abs/2212.06817). That is real-world, well-funded, industrial-scale collection, and it averages out to under 20 usable episodes per robot per day once resets, failures, and curation are counted. On our own rigs we see 20-40 usable tabletop episodes per collection hour on a good day, and 8-15 for long-horizon bimanual work. There is no caching layer for the physical world. Every episode costs wall-clock time on hardware.
The Four Layers of the Bottleneck
The bottleneck is a stack of four constraints, and underestimating any one of them produces a dataset that trains a worse model than its size suggests.
1. Collection throughput. Robots and teleoperators produce data in real time. A 10,000-hour corpus, the scale pi-zero reports pretraining on (arxiv.org/abs/2410.24164), is five years of single-rig operation or a coordinated fleet effort. Throughput is bought with parallel stations, trained operators, and fast reset design, all of which show up in cost per hour.
2. Action-label fidelity. Actions are the labels, and they are only as good as your sync. Continuous-action models (flow matching, diffusion heads) train on action chunks, and timestamp skew between frames and commands degrades them measurably. We hold frame-to-action skew under 10 ms on our rigs because our fine-tune ablations show loss curves separating beyond that.
3. Language annotation quality. An episode without a good instruction is imitation data, not VLA data. Templated labels teach a dozen verbs; natural, diverse instructions teach language conditioning. This layer adds $8-15 per data-hour in our pipeline, and skipping it is the single most common false economy we see.
4. QA and success verification. Failed episodes, teleoperator pauses, dropped frames, and mislabeled successes all pollute training. A 5-10% contamination rate is enough to flatten fine-tune gains on precision tasks, which is why QA is a priced line item and not a virtue.
Why You Cannot Scrape Your Way Out
Web video contains no action labels, so it cannot substitute for the action-labeled core of VLA training. Human video (Ego4D-scale corpora, YouTube) shows what manipulation looks like but never records the motor commands that produced it. The field’s answer, formalized in NVIDIA’s GR00T N1 data pyramid (arxiv.org/abs/2503.14734), is to use web and human video as a broad lower layer for visual priors, synthetic data in the middle, and real teleoperation as the small, expensive, indispensable top layer. The pyramid reduces how much teleop you need. It does not remove the layer.
What the Bottleneck Costs, in Numbers
The cost of VLA training data is best expressed per robot-hour with annotation and QA broken out, because bundled per-episode pricing hides where the money goes. Our production benchmarks:
| Layer | Typical Cost | Failure Mode If Skipped |
|---|---|---|
| Teleop collection (tabletop, VR rig) | $28-38 per robot-hour | None; this is the floor |
| Teleop collection (bimanual, leader-follower) | $40-60 per robot-hour | Under-scoped dexterity data |
| Language annotation (episode-level) | $8-10 per data-hour | Policy ignores instructions |
| Language annotation (segment-level) | $12-15 per data-hour | No long-horizon capability |
| QA + success labels | $5-9 per data-hour | 5-10% contamination, flattened gains |
A 2,000-hour fine-tuning corpus, fully annotated and QA’d, lands between roughly $110k and $170k. That number stops feeling large when compared against the GPU spend it protects: a fine-tune run on contaminated or label-poor data burns the same compute and returns a weaker policy.
How Leading Teams Route Around It
The efficient strategies all share one principle: never pay to collect what transfer can give you, and spend fresh budget on what transfer cannot.
- Start from cross-embodiment pretraining. OpenVLA’s 7B model, pretrained on ~970k OXE episodes, outperformed the 55B RT-2-X on its 29-task evaluation (arxiv.org/abs/2406.09246). The base layer is a solved purchase; open weights exist.
- Co-train with web vision-language data. RT-2 showed this preserves open-vocabulary semantics (arxiv.org/abs/2307.15818). It costs mixture engineering, not collection dollars.
- Buy diversity, not depth. Past roughly 50-150 demonstrations per task, marginal episodes decay in value; Mobile ALOHA reached strong task performance co-training with about 50 demos per task (arxiv.org/abs/2401.02117). New tasks, scenes, objects, and phrasings beat episode 400 of a solved task.
- Match embodiment where it counts. Cross-embodiment transfer is weakest into dexterous hands and humanoids, so reserve embodiment-matched collection for exactly those gaps.
One more route deserves a mention because it is quietly effective: instruction relabeling. DROID and several lab pipelines have used vision-language models to generate or augment instructions on already-collected episodes, which raises language diversity on data you have already paid for. It does not fix bad sync or unlabeled failures, and machine-generated labels need a human QA sample, but as a cost lever it is hard to beat: you are improving the most neglected layer of the stack at annotation prices rather than collection prices.
The common thread in all five strategies is that they treat the bottleneck as a portfolio problem. No single source escapes it. A deliberate stack of open data, web co-training, synthetic coverage, relabeling, and targeted teleoperation gets a team to a working policy at a fraction of the naive all-teleop cost, and every published frontier system is some version of that stack.
We cover the full scoping framework, including the RFP scorecard we make vendors fill out, in our complete guide to VLA training data.
Next Step
if you want to see what clean, sync-verified, language-diverse teleop data looks like before budgeting a collection, download our free LeRobot-format sample pack, or read the full guide first.
Frequently Asked Questions
Why is training data the bottleneck for VLA models instead of compute?
Compute is fungible and rentable; robot demonstrations are produced in real time on physical hardware. The largest open robot corpus holds about one million trajectories, roughly one-thousandth of web vision-language scale, and every additional hour must be physically collected.
How do I scale training data for VLA models on a startup budget?
Pretrain or start from open cross-embodiment checkpoints (OpenVLA, Octo), co-train with web data to keep semantics, and spend collection budget only on embodiment-matched, task-matched fine-tuning data with diverse language labels. Expect $110k-170k for a fully annotated 2,000-hour corpus at current market rates.
Can simulation solve the VLA data bottleneck?
Simulation reduces but does not remove real-data needs. Sim-to-real gaps in contact dynamics and visual realism persist, which is why current pipelines (including GR00T N1) use synthetic data as a middle layer under a real teleoperation layer, not as a replacement.
How many demonstrations per task does a VLA fine-tune need?
The imitation-learning literature repeatedly lands around 50-150 demonstrations per task when fine-tuning on top of a broad pretrained base, with more needed for contact-rich or long-horizon tasks. Diversity across tasks and phrasings matters more than pushing any one task far past that floor.
