Zero robot action labels exist in the world’s largest manipulation video corpora. Thousands of hours of humans pouring, folding, and assembling, on YouTube and in Ego4D-scale datasets, and not one recorded motor command among them. That single number is why the VLA data sourcing debate never resolves cleanly: the cheapest, most scalable source of manipulation data is missing the one field an action model trains on, and every source that has the field costs real money per hour.
So the budget conversation inside most teams collapses into familiar camps. Someone wants to teleoperate everything because actions are the labels. Someone wants to generate everything in simulation because it is cheap. Someone read the GR00T paper and wants to pretrain on YouTube. All three are right about the strengths and quiet about the weaknesses, and the budget only covers one of them properly.
Our thesis, argued below with published results and our own collection benchmarks: these approaches are not substitutes, they are layers, with different label completeness, different transfer properties, and per-hour costs that differ by two orders of magnitude. Treating them as interchangeable is how teams end up with 40,000 synthetic episodes that fine-tune into a policy that cannot close a real gripper on a real deformable object.
This post compares the four practical sourcing approaches: teleoperation, human video, synthetic generation, and open datasets. For each you get what it is, where it wins, where it fails, and what it costs, ending with the decision matrix we use when scoping client collections. Costs marked as DexSet benchmarks are from our own production pipeline; the rest cite primary sources.
Key Takeaways – Teleoperation is the only approach that produces complete observation-action-language triples; it costs $28-60 per robot-hour in our benchmarks. – Human video is the cheapest per hour and the most scalable, but has no action labels; it works as a pretraining layer, never as the whole diet. – Synthetic data has near-zero marginal cost and perfect labels, but pays a sim-to-real tax on contact-rich tasks. – Open datasets (Open X-Embodiment, DROID, BridgeData V2) are effectively free and belong in every mixture; their limit is embodiment and domain mismatch. – The winning pattern in published systems is a pyramid: open + web data at the base, synthetic in the middle, embodiment-matched teleoperation on top.
Approach 1: Teleoperation
Teleoperation data is collected by a human operator directly controlling the robot while every observation and command is recorded, making the robot’s own action stream the training label. This is the approach behind ALOHA/ACT (arxiv.org/abs/2304.13705), DROID (arxiv.org/abs/2403.12945), and the top layer of essentially every serious VLA pipeline including pi-zero’s roughly 10,000-hour corpus (arxiv.org/abs/2410.24164).
Pros: complete and exact action labels; embodiment-matched by construction; supports contact-rich and bimanual tasks; language can be attached at collection time while context is fresh.
Cons: real-time throughput ceiling (20-40 usable tabletop episodes per hour on our rigs, 8-15 for long-horizon work); operator skill variance shows up in the data; rigs cost real money (an ALOHA-class bimanual station is around $20k before cameras).
Cost (DexSet benchmark): $28-38 per robot-hour for VR-rig tabletop collection, $40-60 for leader-follower bimanual work, plus $8-15 per data-hour for language annotation and $5-9 for QA.
Approach 2: Human Egocentric Video
Human video data is footage of people performing tasks, captured from head-mounted or fixed cameras, containing rich visual and behavioral signal but no robot action labels. Ego4D-scale corpora and in-house egocentric capture both fall here.
Pros: an order of magnitude cheaper per hour than teleoperation; humans perform tasks at natural speed with natural strategies; enormous diversity is reachable; head-mounted capture rigs are cheap to scale across many wearers.
Cons: no action labels, so it cannot train the action head directly; embodiment gap between human hands and robot grippers; requires retargeting or representation-learning machinery to become useful.
Cost (DexSet benchmark): $15-25 per hour for structured egocentric capture with task scripts and QA, roughly half the cost of teleoperation for the same wall-clock hour.
Where it fits: as pretraining and co-training signal. NVIDIA’s GR00T N1 explicitly structures this as the wide base of its data pyramid (arxiv.org/abs/2503.14734).
Approach 3: Synthetic and Simulation Data
Synthetic data is generated in physics simulators or by generative models, with actions and annotations produced programmatically at near-zero marginal cost. Perfect labels, infinite resets, no hardware wear.
Pros: marginal cost approaches zero after environment setup; perfect ground truth including states no sensor can capture; safe exploration of failure states; trivially parallelizable.
Cons: the sim-to-real gap is worst exactly where manipulation is hardest: contact dynamics, deformables, friction, specular and transparent objects. Environment engineering is a real up-front cost that budget lines often forget. Policies trained purely in sim routinely need real-data fine-tuning anyway.
Cost: environment and asset engineering is the dominant term (weeks of engineer time per task family); generation itself is compute-priced. Treat it as high fixed cost, near-zero marginal cost.
Approach 4: Open Datasets
Open datasets are published robot demonstration corpora, led by Open X-Embodiment with over one million trajectories across 22 embodiments (arxiv.org/abs/2310.08864), DROID with 76k episodes across 564 scenes, and BridgeData V2 with roughly 60k trajectories (arxiv.org/abs/2308.12952).
Pros: free; instantly available; proven at scale (OpenVLA pretrained on ~970k OXE episodes, arxiv.org/abs/2406.09246); increasingly standardized via LeRobot formats.
Cons: your embodiment, cameras, and tasks are probably not in there; instruction quality varies wildly between constituent datasets; license terms need actual reading; everyone else trains on the same data, so it confers no advantage by itself.
Side-by-Side Comparison
| Dimension | Teleoperation | Human Video | Synthetic | Open Datasets |
|---|---|---|---|---|
| Action labels | Complete, exact | None | Perfect, programmatic | Complete, variable quality |
| Cost per data-hour | $28-60 + annotation (DexSet benchmark) | $15-25 (DexSet benchmark) | Near-zero marginal, high fixed | Free |
| Embodiment match | Exact | Poor (human) | Configurable | Rarely exact |
| Contact-rich fidelity | High | Visual only | Weak to moderate | High where present |
| Scale ceiling | Low (real time) | High | Very high | Fixed (~1M episodes) |
| Language quality | Controllable at source | Must be added | Templated by default | Inconsistent |
| Best role | Fine-tuning top layer | Pretraining base | Mid-layer augmentation | Pretraining base |
The Decision Matrix We Actually Use
The right mixture follows from two questions: how far is your embodiment from the open corpora, and how contact-rich are your target tasks?
- Standard single arm, tabletop tasks: open datasets + a thin teleop fine-tuning layer (200-500 hours). Cheapest viable path.
- Standard arm, contact-rich or precision tasks: add stereo teleop collection on the target tasks; synthetic helps for coverage but will not close the contact gap alone.
- Dexterous hands or humanoids: cross-embodiment transfer is weakest here. Budget for substantial embodiment-matched teleoperation (1,000+ hours) over a human-video and open-data base.
- Long-horizon mobile manipulation: prioritize segment-level language labels; Mobile ALOHA showed strong results co-training with roughly 50 demos per task on a broad base (arxiv.org/abs/2401.02117), so spread hours across many tasks.
Two budgeting notes that surprise first-time buyers. First, annotation and QA scale with data-hours, not with approach, so a cheap collection method with poor labels often costs more per useful training hour than an expensive one delivered clean; a $30 teleop hour that ships with diverse language labels and verified success flags beats a $15 hour that needs a relabeling pass and a 20% cull. Second, the approaches interact: a few hundred hours of well-instrumented teleoperation is also the calibration set that makes your synthetic pipeline honest, because it gives you real contact dynamics to validate against. Teams that sequence it that way (teleop first, sim scaled second) report far less sim-to-real disappointment than teams that generate first and verify never.
The pattern across every row: nobody should pay collection rates for the base of the pyramid, and nobody should expect the free layers to cover the top. For the full scoping framework, mixture ratios, and our vendor RFP scorecard, see the complete guide to VLA training data.
Next Step
if you are weighing these approaches for a real budget, read the complete VLA training data guide or book a demo and we will pressure-test your planned mixture against our collection benchmarks.
Frequently Asked Questions
What is the cheapest way to get training data for a VLA model?
Start with free open datasets (Open X-Embodiment, DROID, BridgeData V2) for pretraining, then buy only embodiment-matched teleoperation for fine-tuning. In our benchmarks that fine-tuning layer costs $28-60 per robot-hour plus annotation, and 200-500 hours is often enough for standard manipulators.
Is synthetic data good enough to train a VLA model?
Not alone. Synthetic data provides cheap coverage and perfect labels but carries a sim-to-real gap that is worst on contact dynamics and deformable objects. Published systems use it as a middle layer between web-scale video and real teleoperation.
How does teleoperation data compare to human video on cost?
In DexSet’s pipeline, structured egocentric human video runs $15-25 per hour versus $28-60 per robot-hour for teleoperation. The difference is labels: teleoperation records executable robot actions, human video does not, so they occupy different layers of the training mixture rather than competing.
Should I collect my own data or use Open X-Embodiment?
Both. OXE-scale open data is the standard pretraining base and costs nothing. Custom collection earns its price where open data is thin: your specific embodiment, your camera geometry, your tasks, and instruction phrasing diverse enough to make language conditioning work.
