Skip to main content

Dexset

dexset
Menu
Request a Demo

Category: Model Training: VLA & Humanoids

5 Hidden Challenges in Training Data for Vision-Language-Action (VLA) Models and How to Solve Them

Posted on by Sainath Gupta

Spot-checking your robot dataset is making things worse. Not the data, your judgment of it: per-episode visual review manufactures confidence, while the defects that actually sink VLA fine-tunes live at the distribution level, where no individual episode looks wrong. That is the thesis of this post, and its practical consequence: the unit of QA for VLA data is the batch and the corpus, not the episode.

The obvious VLA training data quality problems get solved early because they are loud. Not enough episodes, missing camera streams, no language labels: these announce themselves in week one. The problems that actually sink fine-tunes are quiet. They pass a visual spot-check, survive ingestion without errors, and only surface weeks later as a loss curve that plateaus early or a policy that fumbles tasks it should own.

These failures stay hidden for a structural reason. A robot dataset has no compiler. Nothing throws an exception when your timestamps drift 40 ms or when 92% of your instructions share one sentence shape. The data trains. The model just trains worse, and the gap gets attributed to architecture, hyperparameters, or “needing more data,” which leads teams to buy more of the same defective distribution.

We gate every dataset that leaves DexSet through a QA pipeline built around these exact failure modes, so this post is written from the reject pile. For each of the five challenges: what it is, how it hides, how it damages training, and the check that catches it. All five checks are runnable before you spend a single GPU-hour.

Key Takeaways – The five quiet VLA training data quality failures: frame-action timestamp skew, instruction monoculture, silent failure episodes, embodiment/config drift, and mixture starvation. – Each one passes casual inspection and each one measurably degrades fine-tunes; contamination rates as low as 5-10% flatten gains on precision tasks in our ablations. – Every one has a cheap automated check; together they cost $5-9 per data-hour in our pipeline, a fraction of the GPU spend they protect. – Diversity metrics belong in acceptance criteria, not in hopes: template share, verb counts, and per-task hour caps should be contractual.

Challenge 1: Frame-Action Timestamp Skew

Timestamp skew is a systematic offset or drift between when a camera frame was captured and when the recorded action actually executed. Software timestamps on separate device clocks drift, USB cameras buffer unpredictably, and a rig that was aligned in January is often 30-50 ms off by March.

Why it hides: video looks fine, actions replay fine, and nothing is corrupt. Why it hurts: modern VLAs predict action chunks, and continuous-action heads (diffusion in Octo, arxiv.org/abs/2405.12213; flow matching in pi-zero, arxiv.org/abs/2410.24164) regress targets whose alignment with observations is the whole training signal. Skew smears that mapping. In our fine-tune ablations, loss curves separate measurably once skew passes about 10 ms, which is where we set our rig tolerance.

The fix: hardware timestamping where possible, a physical sync event at episode start (a visible gripper snap works), and an automated cross-correlation check between commanded motion and observed motion per episode. Reject on drift, do not average it away.

Challenge 2: Instruction Monoculture

Instruction monoculture is a label distribution where most episodes share a handful of sentence templates, regardless of how many total instructions exist. A 100,000-episode corpus with 14 templates is monoculture at scale; we audited exactly that situation in a recent humanoid client engagement.

Why it hides: ataset cards report episode counts, not instruction diversity – the metric that actually predicts language generalization. Why it hurts: the model learns the templates as fixed keys rather than learning language. RT-1-era templated labels produced policies with a vocabulary of about a dozen verbs (arxiv.org/abs/2212.06817); the semantic breadth in RT-2 came from web co-training, not from the robot labels (arxiv.org/abs/2307.15818). If your labels are monoculture, paraphrase failure is guaranteed.

The fix: measure template share, unique verbs, and spatial-relation coverage per batch, and set hard thresholds. We cap any single template at 8% of a batch and run paraphrase augmentation on failures. Priced honestly, a proper natural-language annotation pass adds $8-15 per data-hour. It is the least skippable line in the budget.

Challenge 3: Silent Failure Episodes

A silent failure episode is a demonstration where the task did not actually succeed – the most damaging robot dataset QA gap in teleoperation-based collection but the episode carries no failure label, so it trains the policy on a wrong outcome presented as a right one. teleoperators drop objects, miss insertions by a few millimeters, or complete the wrong variant – which is why teleoperation data collection requires explicit success criteria baked into the session protocol of an ambiguous task, and at collection pace nobody logs it.

Why it hides: failures look like successes at thumbnail resolution, and per-episode review feels too expensive so teams sample 2% and hope. Why it hurts: imitation learning has no mechanism to discount bad demonstrations; a 5-10% contamination rate is enough to visibly flatten fine-tune gains on precision tasks in our ablations.

The fix: explicit success criteria written per task family before collection, a success label as a required field, and dual review on a stratified sample weighted toward long-horizon and precision tasks. This is most of what our $5-9 per data-hour teleoperation data quality review buys.

Challenge 4: Embodiment and Configuration Drift

Configuration drift is unannounced variation in the physical setup across a collection: cameras nudged between sessions, gripper fingers swapped after wear, firmware updates that change controller latency, tables raised two centimeters. Each change is small. Accumulated, your “single embodiment” corpus is quietly five embodiments.

Why it hides: no single episode is wrong, so episode-level QA passes everything. Why it hurts: the policy either averages across configurations it cannot distinguish or keys on incidental visual cues to tell them apart. Cross-embodiment training works when variation is labeled, which is exactly how Open X-Embodiment makes 22 embodiments useful (arxiv.org/abs/2310.08864); unlabeled variation is just noise.

The fix: a per-session calibration ritual (camera extrinsics check, gripper state, controller version logged into episode metadata) and drift detection that compares session-level statistics. Metadata is free at collection time and unrecoverable afterward.

Challenge 5: Mixture Starvation

Mixture starvation is a training mixture where one data source or task family dominates sampling, starving the components that were the reason you built the mixture. It happens by accident: datasets differ in episode length, so uniform episode sampling silently overweights short-episode sources.

Why it hides: the mixture config looks balanced on paper while the effective batch composition is not. Why it hurts: OpenVLA and Octo both hand-tuned per-dataset mixture weights rather than sampling naively, and OpenVLA’s curated 970k-episode mixture is a large part of how a 7B model beat a 55B one (arxiv.org/abs/2406.09246). Teams that skip mixture sampling accounting also routinely drop web co-training too low and watch open-vocabulary grounding decay.

The fix: account mixtures in timesteps or tokens, not episodes; log effective composition per training run; cap per-task-family hours at collection time so the raw corpus starts balanced (we use a 60-hour cap per family in typical fine-tuning collections).

A note on why these five cluster together: they are all distribution problems rather than instance problems. Any single episode can pass inspection while the corpus as a whole is defective, which is why per-episode spot checks feel reassuring and prove nothing. The unit of QA for VLA data is the batch and the corpus. Once a team internalizes that, the checks below stop looking like overhead and start looking like the cheapest model improvement available.

How Do You Check VLA Training Data Quality? (5 Checks)

ChallengeCheap Detection CheckThreshold We Use
Timestamp skewCommand-vs-observed motion cross-correlation per episodeReject > 10 ms
Instruction monocultureTemplate share + unique-verb count per 1,000-episode batchNo template > 8%
Silent failuresSuccess criteria + stratified dual review5% dual-review floor
Configuration driftSession metadata + session-statistics comparisonAny unlogged change fails
Mixture starvationTimestep-level composition logging per runMatch declared weights ±5%

Run these against any dataset, ours included, before training on it. The full framework, including costs and the vendor RFP scorecard these checks feed into, is in the complete guide to VLA training data.

Next Step

download our free LeRobot-format sample pack and run these five checks against it yourself; the QA report ships alongside the episodes. Or start with the complete VLA training data guide.

Request a Quote

Frequently Asked Questions

What are the most common quality problems in VLA training data?

The quiet ones: frame-action timestamp skew, instruction template monoculture, unlabeled failure episodes, unlogged configuration drift, and unbalanced training mixtures. All five pass casual inspection and all five measurably degrade fine-tunes.

For continuous-action models we hold frame-to-action skew under 10 ms, because our fine-tune ablations show measurable degradation beyond that. Discrete-token models tolerate more, but skew is cheap to control at capture time and expensive to fix later.

Measure instruction template share and unique-verb counts per batch, verify success labels on a stratified dual-reviewed sample, and cross-correlate commanded against observed motion for sync. All three checks run in hours on commodity hardware.

Usually because the new data repeats the old distribution: same tasks, same templates, same scene. Past roughly 50-150 demonstrations per task, marginal episodes add little; audit diversity before adding volume. This is the most common symptom of undetected VLA training data quality problems in the distribution.

Case Study: How We Scaled VLA Training Data for a Humanoid Foundation Model Team

Posted on by Enoch Pakanati

On station four of the collection floor, an operator sets down her leader arms after a clean bowl-to-sink transfer, and the batch dashboard flags the episode anyway. The instruction field reads “put the bowl in the sink,” the same sentence shape as two hundred episodes before it, and the template-share gate has just tripped. That flag, firing in the middle of a live program, is the single most important control in the engagement this post describes.

The client behind that program, a humanoid foundation model team, did not come to us asking for data. They came asking why their model was failing. Their OpenVLA-style fine-tune scored well on standard tabletop evaluations, then collapsed on the bimanual kitchen tasks their product actually needed. Worse, the policy only responded reliably to instructions phrased exactly like the training labels. “Put the bowl in the sink” worked. “Move that bowl into the sink” mostly did not.

The failure had a clean explanation in their data, and it is the most common one we see. Their existing corpus was large by episode count, roughly three times what we ended up collecting, but it covered only 9 task families and used 14 instruction templates. The model had learned those 9 tasks and those 14 sentence shapes, thoroughly, and almost nothing else. Volume was never the problem. Distribution was.

This post walks through what we scoped, what the QA gates caught, what it cost in structure (with our standard benchmark rates), and what changed in their evaluations. If you are planning a fine-tuning collection, the transferable part is the method: diversity targets set as hard gates, not aspirations.

Key Takeaways – 1,400 hours of bimanual teleoperation across 42 task families replaced a corpus 3x larger in episode count but 5x narrower in tasks. – A hard instruction diversity gate (no template above 8% of any batch) fixed paraphrase brittleness; language annotation ran at our standard $12-15 per data-hour, segment-level. – Blending new data 60/40 with the legacy corpus roughly doubled held-out kitchen task success in the client’s own evaluation harness. – QA rejected 11% of collected episodes; the top three causes were teleoperator mid-chunk pauses, frame-action sync drift, and failed-but-unlabeled episodes. – Total program ran 19 weeks from scoping to final delivery in LeRobot-compatible format.

The Starting Point: A Large but Narrow Corpus

A narrow corpus is a dataset whose episode count outruns its task, scene, and language diversity, and it trains models that look strong on in-distribution tests and brittle everywhere else. The client’s legacy data was a textbook case: high volume, 9 task families, 14 templates, single kitchen scene, one camera geometry.

The published evidence already pointed to the fix. Open X-Embodiment showed pooled diversity beating single-domain volume (arxiv.org/abs/2310.08864), and Mobile ALOHA reached strong per-task results co-training with roughly 50 demonstrations per task over a broad base (arxiv.org/abs/2401.02117). Our job was to apply that at commercial fine-tuning scale on their embodiment, where cross-embodiment transfer is weakest.

What We Scoped

The collection specification is the deliverable that matters most, so we fixed it before a single episode was recorded:

  • Volume and spread: 1,400 robot-hours of bimanual teleoperation across 42 task families, capped at 60 hours per family to force breadth.
  • Scenes and objects: 6 kitchen layouts, 300+ object instances, mandated lighting variation across sessions.
  • Capture spec: 4 camera streams per station (two exocentric, head, wrist), 30 Hz, hardware-timestamped, frame-to-action skew held under 10 ms.
  • Language spec: natural free-form instructions written at collection time, segment-level labels with timestamps on all long-horizon episodes, and the diversity gate below.
  • Format: LeRobot-compatible episodes so their training stack ingested data on day one (github.com/huggingface/lerobot).

The Instruction Diversity Gate

An instruction diversity gate is a batch-level QA rule that rejects any 1,000-episode batch where a single instruction template exceeds a set share, in this case 8%. Annotators drift toward comfortable phrasings; without a gate, “pick up the X and place it in the Y” quietly becomes half your labels.

We enforced three thresholds per batch: template share under 8%, minimum unique verb count, and minimum spatial-relation phrase coverage (“behind the cutting board”, “second shelf from the top”). Batches that failed went back for paraphrase augmentation before delivery. This is also where the money went: segment-level annotation with paraphrase passes ran at our standard $12-15 per data-hour, versus $8-10 for a plain episode-level pass. The client initially pushed back on that line item. It ended up being the line item their evaluation results vindicated most directly.

How the Program Ran

A collection program’s timeline is set by station-hours and reset speed, not by calendar ambition, so we planned backwards from throughput. The program ran 19 weeks across six bimanual stations: two weeks of scoping and task-family design with the client’s ML team, one week of operator training and pilot batches, fourteen weeks of production collection, and two weeks of final QA, re-collection of rejected hours, and packaging.

The pilot week matters more than its size suggests. We shipped the first 40 hours to the client immediately and they ran a smoke-test fine-tune before full production started. That surfaced two spec problems while they were still cheap to fix: their training stack expected a different gripper action convention, and one camera’s mounting height put the left workspace edge out of frame on tall-object tasks. Catching either in week twelve instead of week three would have cost hundreds of re-collected hours. If a vendor does not offer a pilot-batch gate before full-rate production, build one into the contract yourself.

Weekly deliveries went out in LeRobot format with per-batch QA reports, so the client’s dataset grew continuously instead of arriving as one end-of-program surprise.

What QA Caught

Episode rejection is the health metric of a collection program, and ours ran at 11% across the engagement. Where those episodes died:

Rejection CauseShare of RejectsWhy It Matters for VLA Training
Teleoperator mid-chunk pauses34%Corrupts action-chunk targets for chunked policies (ACT-style and continuous heads)
Frame-action sync drift > 10 ms27%Measurably degrades continuous-action fine-tunes
Failed episode, unlabeled22%Success mislabeling flattens gains on precision tasks
Dropped frames / camera faults11%Breaks observation continuity
Instruction-episode mismatch6%Directly poisons language conditioning

Every rejected hour was recollected within the fixed budget because our per-hour rates ($40-60 for bimanual leader-follower work, our standard benchmark range) price in an expected rejection band. If a vendor quotes you a rate with no stated rejection policy, that risk is sitting on your side of the table.

Results

The client fine-tuned on the new corpus blended 60/40 with their legacy data and ran their internal evaluation harness before and after. Directional results, their harness, not an audited benchmark:

  • Held-out kitchen task success roughly doubled against the legacy-data baseline.
  • Paraphrased-instruction success moved from near-random to parity with template phrasing.
  • Long-horizon task completion improved most on tasks with segment-level labels, consistent with what the pi-zero pipeline reports about post-training data quality (arxiv.org/abs/2410.24164).

The result we quote most often is the second one, because it isolates the language layer. Nothing about the model changed. The labels changed.

What Transfers to Your Program

Four rules from this engagement apply to almost any VLA fine-tuning collection:

  • Audit distribution before buying volume; count task families and unique templates in your existing corpus first.
  • Set diversity thresholds as batch-level rejection gates, not guidelines.
  • Price annotation and QA as separate line items so you can see what you are cutting when you cut.
  • Cap hours per task family; the marginal episode of a covered task is your most overpriced purchase.

The full scoping framework, cost tables, and the RFP scorecard we build these programs from are in the complete guide to VLA training data.

Next Step

if your evaluations look like this client’s before-picture, book a scoping call. We will audit your corpus distribution before recommending a single collection hour, or start with the complete VLA training data guide.

Request a Quote

Frequently Asked Questions

How many hours of data does it take to fine-tune a VLA for a humanoid?

This program used 1,400 hours across 42 task families, blended with existing data. Humanoids sit at the expensive end because cross-embodiment transfer from open datasets is weakest for them; standard single arms often need only 200-500 hours.

A batch-level QA rule that rejects annotation batches where any single instruction template exceeds a set share (we used 8%), with minimum thresholds for unique verbs and spatial-relation phrases. It prevents the label monoculture that makes VLA policies brittle to paraphrasing.

The legacy data still carried valid task knowledge; the 60/40 blend kept that signal while the new data corrected the distribution. Full replacement wastes paid-for coverage and can regress previously working tasks.

We ran 11% on this program, and 10-15% is the band we consider healthy for bimanual long-horizon work. A vendor reporting near-zero rejection is usually not checking sync, pauses, or success labels.

Comparing Training Data Approaches for Vision-Language-Action (VLA) Models: Pros, Cons & Costs

Posted on by Sainath Gupta

Zero robot action labels exist in the world’s largest manipulation video corpora. Thousands of hours of humans pouring, folding, and assembling, on YouTube and in Ego4D-scale datasets, and not one recorded motor command among them. That single number is why the VLA data sourcing debate never resolves cleanly: the cheapest, most scalable source of manipulation data is missing the one field an action model trains on, and every source that has the field costs real money per hour.

So the budget conversation inside most teams collapses into familiar camps. Someone wants to teleoperate everything because actions are the labels. Someone wants to generate everything in simulation because it is cheap. Someone read the GR00T paper and wants to pretrain on YouTube. All three are right about the strengths and quiet about the weaknesses, and the budget only covers one of them properly.

Our thesis, argued below with published results and our own collection benchmarks: these approaches are not substitutes, they are layers, with different label completeness, different transfer properties, and per-hour costs that differ by two orders of magnitude. Treating them as interchangeable is how teams end up with 40,000 synthetic episodes that fine-tune into a policy that cannot close a real gripper on a real deformable object.

This post compares the four practical sourcing approaches: teleoperation, human video, synthetic generation, and open datasets. For each you get what it is, where it wins, where it fails, and what it costs, ending with the decision matrix we use when scoping client collections. Costs marked as DexSet benchmarks are from our own production pipeline; the rest cite primary sources.

Key Takeaways – Teleoperation is the only approach that produces complete observation-action-language triples; it costs $28-60 per robot-hour in our benchmarks. – Human video is the cheapest per hour and the most scalable, but has no action labels; it works as a pretraining layer, never as the whole diet. – Synthetic data has near-zero marginal cost and perfect labels, but pays a sim-to-real tax on contact-rich tasks. – Open datasets (Open X-Embodiment, DROID, BridgeData V2) are effectively free and belong in every mixture; their limit is embodiment and domain mismatch. – The winning pattern in published systems is a pyramid: open + web data at the base, synthetic in the middle, embodiment-matched teleoperation on top.

Approach 1: Teleoperation

Teleoperation data is collected by a human operator directly controlling the robot while every observation and command is recorded, making the robot’s own action stream the training label. This is the approach behind ALOHA/ACT (arxiv.org/abs/2304.13705), DROID (arxiv.org/abs/2403.12945), and the top layer of essentially every serious VLA pipeline including pi-zero’s roughly 10,000-hour corpus (arxiv.org/abs/2410.24164).

Pros: complete and exact action labels; embodiment-matched by construction; supports contact-rich and bimanual tasks; language can be attached at collection time while context is fresh.

Cons: real-time throughput ceiling (20-40 usable tabletop episodes per hour on our rigs, 8-15 for long-horizon work); operator skill variance shows up in the data; rigs cost real money (an ALOHA-class bimanual station is around $20k before cameras).

Cost (DexSet benchmark): $28-38 per robot-hour for VR-rig tabletop collection, $40-60 for leader-follower bimanual work, plus $8-15 per data-hour for language annotation and $5-9 for QA.

Approach 2: Human Egocentric Video

Human video data is footage of people performing tasks, captured from head-mounted or fixed cameras, containing rich visual and behavioral signal but no robot action labels. Ego4D-scale corpora and in-house egocentric capture both fall here.

Pros: an order of magnitude cheaper per hour than teleoperation; humans perform tasks at natural speed with natural strategies; enormous diversity is reachable; head-mounted capture rigs are cheap to scale across many wearers.

Cons: no action labels, so it cannot train the action head directly; embodiment gap between human hands and robot grippers; requires retargeting or representation-learning machinery to become useful.

Cost (DexSet benchmark): $15-25 per hour for structured egocentric capture with task scripts and QA, roughly half the cost of teleoperation for the same wall-clock hour.

Where it fits: as pretraining and co-training signal. NVIDIA’s GR00T N1 explicitly structures this as the wide base of its data pyramid (arxiv.org/abs/2503.14734).

Approach 3: Synthetic and Simulation Data

Synthetic data is generated in physics simulators or by generative models, with actions and annotations produced programmatically at near-zero marginal cost. Perfect labels, infinite resets, no hardware wear.

Pros: marginal cost approaches zero after environment setup; perfect ground truth including states no sensor can capture; safe exploration of failure states; trivially parallelizable.

Cons: the sim-to-real gap is worst exactly where manipulation is hardest: contact dynamics, deformables, friction, specular and transparent objects. Environment engineering is a real up-front cost that budget lines often forget. Policies trained purely in sim routinely need real-data fine-tuning anyway.

Cost: environment and asset engineering is the dominant term (weeks of engineer time per task family); generation itself is compute-priced. Treat it as high fixed cost, near-zero marginal cost.

Approach 4: Open Datasets

Open datasets are published robot demonstration corpora, led by Open X-Embodiment with over one million trajectories across 22 embodiments (arxiv.org/abs/2310.08864), DROID with 76k episodes across 564 scenes, and BridgeData V2 with roughly 60k trajectories (arxiv.org/abs/2308.12952).

Pros: free; instantly available; proven at scale (OpenVLA pretrained on ~970k OXE episodes, arxiv.org/abs/2406.09246); increasingly standardized via LeRobot formats.

Cons: your embodiment, cameras, and tasks are probably not in there; instruction quality varies wildly between constituent datasets; license terms need actual reading; everyone else trains on the same data, so it confers no advantage by itself.

Side-by-Side Comparison

DimensionTeleoperationHuman VideoSyntheticOpen Datasets
Action labelsComplete, exactNonePerfect, programmaticComplete, variable quality
Cost per data-hour$28-60 + annotation (DexSet benchmark)$15-25 (DexSet benchmark)Near-zero marginal, high fixedFree
Embodiment matchExactPoor (human)ConfigurableRarely exact
Contact-rich fidelityHighVisual onlyWeak to moderateHigh where present
Scale ceilingLow (real time)HighVery highFixed (~1M episodes)
Language qualityControllable at sourceMust be addedTemplated by defaultInconsistent
Best roleFine-tuning top layerPretraining baseMid-layer augmentationPretraining base

The Decision Matrix We Actually Use

The right mixture follows from two questions: how far is your embodiment from the open corpora, and how contact-rich are your target tasks?

  • Standard single arm, tabletop tasks: open datasets + a thin teleop fine-tuning layer (200-500 hours). Cheapest viable path.
  • Standard arm, contact-rich or precision tasks: add stereo teleop collection on the target tasks; synthetic helps for coverage but will not close the contact gap alone.
  • Dexterous hands or humanoids: cross-embodiment transfer is weakest here. Budget for substantial embodiment-matched teleoperation (1,000+ hours) over a human-video and open-data base.
  • Long-horizon mobile manipulation: prioritize segment-level language labels; Mobile ALOHA showed strong results co-training with roughly 50 demos per task on a broad base (arxiv.org/abs/2401.02117), so spread hours across many tasks.

Two budgeting notes that surprise first-time buyers. First, annotation and QA scale with data-hours, not with approach, so a cheap collection method with poor labels often costs more per useful training hour than an expensive one delivered clean; a $30 teleop hour that ships with diverse language labels and verified success flags beats a $15 hour that needs a relabeling pass and a 20% cull. Second, the approaches interact: a few hundred hours of well-instrumented teleoperation is also the calibration set that makes your synthetic pipeline honest, because it gives you real contact dynamics to validate against. Teams that sequence it that way (teleop first, sim scaled second) report far less sim-to-real disappointment than teams that generate first and verify never.

The pattern across every row: nobody should pay collection rates for the base of the pyramid, and nobody should expect the free layers to cover the top. For the full scoping framework, mixture ratios, and our vendor RFP scorecard, see the complete guide to VLA training data.

Next Step

if you are weighing these approaches for a real budget, read the complete VLA training data guide or book a demo and we will pressure-test your planned mixture against our collection benchmarks.

Request a Quote

Frequently Asked Questions

What is the cheapest way to get training data for a VLA model?

Start with free open datasets (Open X-Embodiment, DROID, BridgeData V2) for pretraining, then buy only embodiment-matched teleoperation for fine-tuning. In our benchmarks that fine-tuning layer costs $28-60 per robot-hour plus annotation, and 200-500 hours is often enough for standard manipulators.

Not alone. Synthetic data provides cheap coverage and perfect labels but carries a sim-to-real gap that is worst on contact dynamics and deformable objects. Published systems use it as a middle layer between web-scale video and real teleoperation.

In DexSet’s pipeline, structured egocentric human video runs $15-25 per hour versus $28-60 per robot-hour for teleoperation. The difference is labels: teleoperation records executable robot actions, human video does not, so they occupy different layers of the training mixture rather than competing.

Both. OXE-scale open data is the standard pretraining base and costs nothing. Custom collection earns its price where open data is thin: your specific embodiment, your camera geometry, your tasks, and instruction phrasing diverse enough to make language conditioning work.

Why Training Data for Vision-Language-Action (VLA) Models Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

Somewhere on every VLA data plan sits a line that reads “annotation and QA,” and it is the checklist item nearly everyone skips when the budget gets real. Collection hours get scoped to the dollar. GPU spend gets scoped to the node. The layers that decide whether those hours train anything get a checkbox. That skipped line is usually where we find the explanation when a team arrives with a Prismatic-style backbone, sensible action head, clean training code, and a model that still cannot pick up a mug it has never seen when the instruction is phrased slightly differently.

The deeper reason is structural, not a bug. Language models got to scrape a web that humanity spent thirty years writing. VLA models need trajectories where a physical robot moved and every action was recorded, and that data did not exist until someone paid to create it, one teleoperated episode at a time. Open X-Embodiment, the largest pooled open corpus, holds just over one million trajectories from 22 embodiments (arxiv.org/abs/2310.08864). Web-scale image-text corpora hold billions of pairs. Physical AI is trying to reach foundation-model behavior on roughly one-thousandth of the data its sibling fields consider table stakes.

This post argues one thesis: the VLA data bottleneck is not a single constraint but a four-layer stack (collection throughput, action-label fidelity, language annotation, QA), and teams that budget only for the first layer buy datasets that train worse models than their size suggests. We show where each layer binds, what it costs to push against, and which strategies the leading labs use to spend around it efficiently. We collect and QA this data for a living at DexSet, so the cost numbers are our own production benchmarks, flagged as such.

Key Takeaways – The robot data gap is around three orders of magnitude versus web vision-language corpora, and it cannot be closed by scraping. – The bottleneck has four layers: collection throughput, action-label fidelity, language annotation quality, and QA. Most teams only budget for the first. – RT-1 took 17 months and 13 robots to produce ~130k episodes; that throughput ceiling is why teleoperation economics matter. – In our benchmarks, usable teleop data costs $28-60 per robot-hour to collect, plus $8-15 per hour for language annotation and $5-9 for QA. – The efficient escape routes are cross-embodiment pretraining, web co-training, and spending fresh budget on diversity rather than raw episode count.

What Makes VLA Training Data a Bottleneck?

The VLA data bottleneck is the mismatch between how fast robot demonstration data can be produced and how much of it foundation-model training consumes. Every VLA needs episodes containing synchronized observations, executed actions, and language instructions, and each of those three elements has its own production constraint.

Consider the throughput math. RT-1’s dataset took a fleet of 13 robots 17 months to reach roughly 130k episodes across about 700 tasks (arxiv.org/abs/2212.06817). That is real-world, well-funded, industrial-scale collection, and it averages out to under 20 usable episodes per robot per day once resets, failures, and curation are counted. On our own rigs we see 20-40 usable tabletop episodes per collection hour on a good day, and 8-15 for long-horizon bimanual work. There is no caching layer for the physical world. Every episode costs wall-clock time on hardware.

The Four Layers of the Bottleneck

The bottleneck is a stack of four constraints, and underestimating any one of them produces a dataset that trains a worse model than its size suggests.

1. Collection throughput. Robots and teleoperators produce data in real time. A 10,000-hour corpus, the scale pi-zero reports pretraining on (arxiv.org/abs/2410.24164), is five years of single-rig operation or a coordinated fleet effort. Throughput is bought with parallel stations, trained operators, and fast reset design, all of which show up in cost per hour.

2. Action-label fidelity. Actions are the labels, and they are only as good as your sync. Continuous-action models (flow matching, diffusion heads) train on action chunks, and timestamp skew between frames and commands degrades them measurably. We hold frame-to-action skew under 10 ms on our rigs because our fine-tune ablations show loss curves separating beyond that.

3. Language annotation quality. An episode without a good instruction is imitation data, not VLA data. Templated labels teach a dozen verbs; natural, diverse instructions teach language conditioning. This layer adds $8-15 per data-hour in our pipeline, and skipping it is the single most common false economy we see.

4. QA and success verification. Failed episodes, teleoperator pauses, dropped frames, and mislabeled successes all pollute training. A 5-10% contamination rate is enough to flatten fine-tune gains on precision tasks, which is why QA is a priced line item and not a virtue.

Why You Cannot Scrape Your Way Out

Web video contains no action labels, so it cannot substitute for the action-labeled core of VLA training. Human video (Ego4D-scale corpora, YouTube) shows what manipulation looks like but never records the motor commands that produced it. The field’s answer, formalized in NVIDIA’s GR00T N1 data pyramid (arxiv.org/abs/2503.14734), is to use web and human video as a broad lower layer for visual priors, synthetic data in the middle, and real teleoperation as the small, expensive, indispensable top layer. The pyramid reduces how much teleop you need. It does not remove the layer.

What the Bottleneck Costs, in Numbers

The cost of VLA training data is best expressed per robot-hour with annotation and QA broken out, because bundled per-episode pricing hides where the money goes. Our production benchmarks:

LayerTypical CostFailure Mode If Skipped
Teleop collection (tabletop, VR rig)$28-38 per robot-hourNone; this is the floor
Teleop collection (bimanual, leader-follower)$40-60 per robot-hourUnder-scoped dexterity data
Language annotation (episode-level)$8-10 per data-hourPolicy ignores instructions
Language annotation (segment-level)$12-15 per data-hourNo long-horizon capability
QA + success labels$5-9 per data-hour5-10% contamination, flattened gains

A 2,000-hour fine-tuning corpus, fully annotated and QA’d, lands between roughly $110k and $170k. That number stops feeling large when compared against the GPU spend it protects: a fine-tune run on contaminated or label-poor data burns the same compute and returns a weaker policy.

How Leading Teams Route Around It

The efficient strategies all share one principle: never pay to collect what transfer can give you, and spend fresh budget on what transfer cannot.

  • Start from cross-embodiment pretraining. OpenVLA’s 7B model, pretrained on ~970k OXE episodes, outperformed the 55B RT-2-X on its 29-task evaluation (arxiv.org/abs/2406.09246). The base layer is a solved purchase; open weights exist.
  • Co-train with web vision-language data. RT-2 showed this preserves open-vocabulary semantics (arxiv.org/abs/2307.15818). It costs mixture engineering, not collection dollars.
  • Buy diversity, not depth. Past roughly 50-150 demonstrations per task, marginal episodes decay in value; Mobile ALOHA reached strong task performance co-training with about 50 demos per task (arxiv.org/abs/2401.02117). New tasks, scenes, objects, and phrasings beat episode 400 of a solved task.
  • Match embodiment where it counts. Cross-embodiment transfer is weakest into dexterous hands and humanoids, so reserve embodiment-matched collection for exactly those gaps.

One more route deserves a mention because it is quietly effective: instruction relabeling. DROID and several lab pipelines have used vision-language models to generate or augment instructions on already-collected episodes, which raises language diversity on data you have already paid for. It does not fix bad sync or unlabeled failures, and machine-generated labels need a human QA sample, but as a cost lever it is hard to beat: you are improving the most neglected layer of the stack at annotation prices rather than collection prices.

The common thread in all five strategies is that they treat the bottleneck as a portfolio problem. No single source escapes it. A deliberate stack of open data, web co-training, synthetic coverage, relabeling, and targeted teleoperation gets a team to a working policy at a fraction of the naive all-teleop cost, and every published frontier system is some version of that stack.

We cover the full scoping framework, including the RFP scorecard we make vendors fill out, in our complete guide to VLA training data.

Next Step

if you want to see what clean, sync-verified, language-diverse teleop data looks like before budgeting a collection, download our free LeRobot-format sample pack, or read the full guide first.

Request a Quote

Frequently Asked Questions

Why is training data the bottleneck for VLA models instead of compute?

Compute is fungible and rentable; robot demonstrations are produced in real time on physical hardware. The largest open robot corpus holds about one million trajectories, roughly one-thousandth of web vision-language scale, and every additional hour must be physically collected.

Pretrain or start from open cross-embodiment checkpoints (OpenVLA, Octo), co-train with web data to keep semantics, and spend collection budget only on embodiment-matched, task-matched fine-tuning data with diverse language labels. Expect $110k-170k for a fully annotated 2,000-hour corpus at current market rates.

Simulation reduces but does not remove real-data needs. Sim-to-real gaps in contact dynamics and visual realism persist, which is why current pipelines (including GR00T N1) use synthetic data as a middle layer under a real teleoperation layer, not as a replacement.

The imitation-learning literature repeatedly lands around 50-150 demonstrations per task when fine-tuning on top of a broad pretrained base, with more needed for contact-rich or long-horizon tasks. Diversity across tasks and phrasings matters more than pushing any one task far past that floor.

The Complete Guide to Training Data for Vision-Language-Action (VLA) Models (2026)

Posted on by Sainath Gupta

Five billion image-text pairs sit in LAION-5B (arxiv.org/abs/2210.08402); just over one million robot trajectories sit in Open X-Embodiment, the largest open robot dataset ever assembled (arxiv.org/abs/2310.08864). Those two numbers should not both describe the training diet of a single model family, yet every production VLA pairs a vision-language half fed at web scale with an action half starved at roughly one-thousandth of it. That mismatch, more than any architecture choice, is the defining constraint of physical AI in 2026.

The gap exists because the two halves come from different worlds. The vision-language half was pretrained on image-text pairs that already existed. The action half needs robot trajectories that do not exist until someone physically collects them, frame-synchronized, action-labeled, and paired with language instructions. You cannot scrape your way out of that. Which is why, on most VLA programs, the architecture question gets settled in a week and the compute budget in a month, while the question of how many language-annotated trajectories the model needs, at what task diversity, on which embodiments, goes unanswered the longest.

This guide gives you the full picture: what VLA training data is, how the landmark models (RT-1, RT-2, OpenVLA, Octo, pi-zero, GR00T) actually sourced their data, how action representation changes your labeling requirements, what collection and annotation cost per hour, and how to scope a dataset that transfers to your embodiment instead of overfitting to someone else’s lab. The thesis we argue throughout, and the one our delivery data keeps confirming: once you stand on the open-data base layer, dataset composition (task diversity, instruction diversity, capture fidelity) predicts VLA capability better than episode count, so budget should follow distribution, not volume.

DexSet supplies egocentric, exocentric, teleoperation, mono, and stereo data to robot foundation model teams. The numbers in this guide come from our own rigs, QA pipelines, and cost benchmarks, alongside the primary papers. Where a figure is our internal benchmark rather than a published result, we say so.

TL;DR: Key Takeaways – VLA training data consists of robot trajectories (synchronized camera frames + robot actions) paired with natural-language instructions, usually collected via teleoperation. – Landmark scale points: RT-1 used ~130k episodes over 17 months; OpenVLA pretrained on ~970k episodes from Open X-Embodiment; pi-zero reports on the order of 10,000 hours of dexterous robot data. – Action representation drives labeling spec: discrete action tokens (RT-2, OpenVLA) tolerate lower control rates; continuous-action heads (pi-zero flow matching, Octo diffusion) reward 50 Hz, low-jitter capture. – In our benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity; a proper language annotation pass adds $8-15 per hour of data. – Data mixture beats raw volume: co-training on web-scale vision-language data plus a well-balanced robot mixture is what gives VLAs semantic generalization. – Task and scene diversity predict generalization better than episode count once you pass a per-task floor (roughly 50-150 demonstrations per task in the imitation-learning literature).

What Is Training Data for Vision-Language-Action (VLA) Models?

Training data for VLA models is a corpus of robot demonstration episodes in which each timestep pairs visual observations, robot proprioceptive state, and executed actions, and each episode (or segment) carries a natural-language instruction describing the task. A VLA model consumes images and an instruction as input and produces robot actions as output, so its training data must contain all three aligned in time.

A single well-formed episode contains:

  • Observations. One or more camera streams (wrist-mounted, over-the-shoulder exocentric, or head-mounted egocentric), typically 224×224 to 1080p RGB at 10-60 fps, optionally stereo pairs or depth.
  • Proprioception. Joint positions, gripper state, end-effector pose, usually at the control rate.
  • Actions. The command stream that produced the motion: delta end-effector poses, joint targets, or gripper commands, timestamped against frames.
  • Language. An instruction such as “put the red mug on the top shelf,” attached at episode level, and increasingly at sub-episode segment level for long-horizon tasks.

The entity chain matters for anyone mapping this space: teleoperation rigs (ALOHA-style leader-follower arms, VR controllers, exoskeletons) produce demonstrations; demonstrations feed imitation learning; imitation learning at scale, on top of a pretrained vision-language backbone, is what turns a VLM into a VLA. The Hugging Face LeRobot project has become the de facto open format for storing these episodes, and its dataset cards are a good reference for what a clean schema looks like.

How the Landmark VLA Models Sourced Their Data

Every major VLA result is downstream of a specific, documented data strategy, and the strategies differ more than the architectures do. Reading the data sections of these papers teaches you more about building a VLA than reading the model sections.

Model (Year)OrgParamsRobot DataLanguage SourceAction Representation
RT-1 (2022)Google35M~130k episodes, 13 robots, 17 months, ~700 tasksTemplated instructionsDiscrete tokens, 256 bins (arxiv.org/abs/2212.06817)
RT-2 (2023)Google DeepMind12B / 55BRT-1 data, co-fine-tuned with web-scale VLM dataTemplated + web textActions as text tokens (arxiv.org/abs/2307.15818)
Octo (2024)UC Berkeley et al.27M / 93M800k trajectories from Open X-EmbodimentMixed, dataset-dependentDiffusion head, continuous (arxiv.org/abs/2405.12213)
OpenVLA (2024)Stanford et al.7B970k episodes from Open X-EmbodimentMixed OXE instructionsDiscrete tokens (arxiv.org/abs/2406.09246)
pi-zero (2024)Physical Intelligence~3BOn the order of 10k hours, 7 platforms, 68 task familiesNatural + segment-level relabelingFlow matching, 50 Hz continuous (arxiv.org/abs/2410.24164)
GR00T N1 (2025)NVIDIA2BData pyramid: web video + synthetic + real robot dataMixed human + generatedContinuous, diffusion-based (arxiv.org/abs/2503.14734)

Three lessons sit in that table. First, scale climbed from 130k episodes to roughly a million in two years, but the biggest capability jumps (RT-2 over RT-1, pi-zero over Octo-class models) came from mixture changes, not just volume. Second, OpenVLA at 7B outperformed the 55B RT-2-X on the paper’s 29-task evaluation suite, which tells you curated cross-embodiment data can substitute for an order of magnitude of parameters. Third, nobody serious trains on robot data alone anymore; co-training against web-scale vision-language corpora is standard because it is what keeps semantic knowledge (what “the leftmost ripe banana” means) alive while the model learns motor control.

Core Data Modalities for VLA Training

Modality choice determines what your VLA can learn and what your collection rig must capture. The main axes are viewpoint (egocentric vs exocentric), depth strategy (mono vs stereo), and collection method (teleoperation vs human video).

Egocentric vs Exocentric Views

Egocentric data is captured from the robot’s or demonstrator’s own viewpoint; exocentric data is captured from external cameras watching the workspace. Wrist and head cameras give the policy view-invariant, occlusion-realistic inputs and tend to transfer better across scene layouts. Exocentric views give annotators and QA reviewers the context to verify task success and label segments. In our pipelines we treat exocentric streams as partly a QA instrument: an episode with only a wrist view is roughly twice as expensive to verify because reviewers must scrub proprioception logs to confirm success.

Mono vs Stereo

Mono capture records a single RGB stream per viewpoint; stereo capture records calibrated pairs that support depth estimation. Most published VLAs train on mono RGB because the backbone VLMs are mono. Stereo earns its extra cost on contact-rich and clutter-heavy manipulation, where depth ambiguity is a dominant failure mode. Our rule of thumb: mono for pretraining breadth, stereo for the fine-tuning set on your target tasks.

Teleoperation vs Human Video

Teleoperation produces the gold standard because the action stream is the robot’s own. Leader-follower rigs like ALOHA (arxiv.org/abs/2304.13705) hit high dexterity at a hardware cost around $20k per station; VR-based rigs are cheaper and faster to train operators on but add pose-retargeting noise. Human egocentric video (Ego4D, EgoExo4D scale) has no action labels, so it enters VLA training as a co-training or pretraining signal, the approach GR00T N1 formalizes as a data pyramid. Human video buys you visual and behavioral priors cheaply; it never replaces the teleop layer where actions live.

Action Representation: Tokenization vs Continuous Actions

Action representation is the choice of how robot actions are encoded for the model to predict, and it silently rewrites your data spec. There are two dominant families.

Discrete action tokenization (RT-1, RT-2, OpenVLA) bins each action dimension, typically into 256 buckets, and treats action prediction as next-token prediction. It plugs directly into a language-model backbone and tolerates modest control rates. The FAST tokenizer (arxiv.org/abs/2501.09747) pushed this further with a compression-based scheme that makes autoregressive VLAs viable on high-frequency data.

Continuous action heads (Octo’s diffusion head, pi-zero’s flow matching) regress action chunks directly. They excel at dexterous, high-rate control (pi-zero runs 50 Hz) but they punish sloppy capture. Timestamp jitter between frames and actions that a tokenized model shrugs off will smear a flow-matching target distribution.

What this means for data buyers, from our QA logs:

  • For tokenized targets, 10-15 Hz aligned capture is usually sufficient; invest the savings in task diversity.
  • For continuous heads, capture at 30-60 Hz with hardware-timestamped sync; we hold frame-to-action skew under 10 ms on our teleop rigs because beyond that we can measure it in fine-tune loss.
  • Action-chunk prediction (introduced with ACT on ALOHA) means episodes must be free of teleoperator pauses mid-chunk, which is a QA check most teams add only after their first bad fine-tune.

Language Annotation: The Layer Everyone Underestimates

Language annotation is the process of attaching instructions to robot episodes, and it is where VLA datasets most often quietly fail. A VLA is only as language-conditioned as its labels are diverse.

Templated vs natural instructions. RT-1-era datasets used templates (“pick X”, “move X near Y”). Templates are cheap and consistent, and they teach the model a grammar of about a dozen verbs. Models trained on templated instructions follow paraphrases poorly; the semantic generalization RT-2 demonstrated came from the web co-training, not the robot labels. Current practice, visible in the pi-zero and DROID (arxiv.org/abs/2403.12945) pipelines, is natural free-form instructions plus paraphrase augmentation.

Episode-level vs segment-level labeling. Episode-level labeling attaches one instruction to a whole demonstration. Segment-level labeling breaks a long-horizon episode into sub-tasks (“open the drawer”, “place the spoon inside”, “close the drawer”) with aligned timestamps. Segment labels cost more but unlock long-horizon training and better credit assignment. In our pipeline, segment-level labeling costs roughly 2.5x the episode-level pass because annotators must scrub video and mark boundaries, not just watch and describe.

Our first-hand benchmark: a language annotation pass adds $8-15 per hour of robot data depending on instruction density and whether we run single or dual annotation. We also run an instruction diversity QA gate: across any 1,000-episode batch we require minimum thresholds for unique verbs, unique object references, and spatial-relation phrases, because we have watched fine-tunes on label-poor batches lose the ability to follow anything but the majority phrasing.

Data Mixtures, Co-Training, and Cross-Embodiment Transfer

A data mixture is the weighted combination of datasets a VLA trains on, and mixture design is now a first-class engineering discipline. Three decisions dominate.

Web co-training ratio. RT-2 established that co-fine-tuning on web vision-language data alongside robot data preserves semantic breadth. Teams that fine-tune on robot data alone consistently report the model “forgetting” open-vocabulary grounding.

Cross-embodiment weighting. Open X-Embodiment showed that pooling data across 22 robot types improves average success, and OpenVLA and Octo both hand-tuned per-dataset mixture weights rather than sampling uniformly. Transfer across embodiments is real but not free: it appears strongest between similar morphologies (single-arm manipulators with parallel grippers) and weakest into dexterous hands and humanoids, which is exactly why humanoid programs still commission embodiment-matched collection.

There is no published universal mixture ratio, and be suspicious of anyone who quotes one. What the papers do agree on is the shape of the curve: heavily downweight any single dataset that would dominate by timestep count, keep a meaningful web co-training fraction throughout fine-tuning rather than dropping it to zero, and upweight the small datasets that match your target embodiment and camera geometry. In practice we see teams land between 10% and 40% embodiment-matched data by timestep in their final fine-tuning mixtures, with the exact point set by held-out evaluation rather than by principle. Measure mixtures in timesteps or tokens, not episodes; episode lengths vary enough between sources that episode-level accounting quietly misstates composition by 2-3x.

Diversity vs depth. Once a task passes a demonstration floor (Mobile ALOHA showed strong results co-training with as few as 50 demonstrations per task on top of a broad base, arxiv.org/abs/2401.02117), the marginal episode of task 12 is worth less than the first episode of task 13. When we rebalance client collections, we usually shift budget from episode count toward scene, object, and instruction diversity, and the downstream evaluations reward it.

Cost and Economics: What VLA Training Data Actually Costs

VLA data cost is driven by four stacked line items: rig amortization, operator time, annotation, and QA. Vendors who quote a single opaque per-episode price are hiding this stack. Here is ours, from DexSet production benchmarks (round figures, typical ranges we see):

Cost ComponentTypical RangeNotes
Teleop collection (VR rig, tabletop tasks)$28-38 per robot-hourIncludes operator, station amortization, resets
Teleop collection (leader-follower, bimanual/contact-rich)$40-60 per robot-hourALOHA-class rigs, slower resets, higher operator skill
Language annotation, episode-level$8-10 per data-hourNatural instructions, single pass
Language annotation, segment-level + paraphrases$12-15 per data-hourTimestamped sub-task boundaries, dual pass on 10%
QA and success verification$5-9 per data-hourSuccess labels, sync checks, instruction diversity gate
Useful episodes per collection hour20-40 tabletop; 8-15 long-horizonAfter discarding failed/aborted episodes

Worked example: a 2,000-hour bimanual manipulation corpus with segment-level language and full QA lands between $114k and $168k. Against that, note that pi-zero-class results were built on roughly 10,000 hours. This is why mixture design and cross-embodiment pretraining matter commercially: nobody fine-tuning a foundation model should be paying to recollect the base layer that OXE-scale open data already provides. Pay for the embodiment-matched, task-matched, well-labeled top of the pyramid.

Case Study Proof: Scaling a Fine-Tuning Corpus for a Humanoid Team

A humanoid foundation model team came to us with an OpenVLA-style base model and a fine-tuning problem: strong tabletop scores, weak performance on their bimanual kitchen tasks, and instructions that only worked when phrased like the training templates. We scoped a 1,400-hour teleoperation collection across 42 task families with a hard instruction-diversity requirement (no template allowed to exceed 8% of a batch) and segment-level labels on all long-horizon episodes.

The mixture, not the volume, did the work. Their prior corpus had 3x our episode count but 9 task families and 14 instruction templates. After fine-tuning on the new corpus blended 60/40 with their legacy data, their internal evaluation showed task success on held-out kitchen tasks roughly doubling, and paraphrased-instruction success moved from near-random to parity with template phrasing. We report those as directional results from the client’s own evaluation harness, not audited benchmarks. The full breakdown is in our case study write-up.

Scoping Your Own Collection: The RFP Scorecard

A VLA data RFP scorecard is a structured checklist for comparing data vendors on the dimensions that predict fine-tune quality. If you take one action from this guide, make vendors answer these in writing:

  • Frame-to-action sync tolerance, in milliseconds, with measurement method.
  • Control and capture rates, and whether timestamps are hardware or software.
  • Instruction sourcing: templated, natural, or paraphrase-augmented, with a diversity metric per batch.
  • Episode-level vs segment-level labeling, priced separately.
  • Success-labeling protocol and inter-annotator agreement.
  • Cost per robot-hour, broken into collection, annotation, and QA. Refuse bundled per-episode pricing.
  • Delivery format (LeRobot-compatible schemas save your team weeks).

We publish ours as a downloadable template. [Download the VLA Data RFP Scorecard] and use it against us too.

Related reading this week:

Next Step

If you are scoping a VLA data collection this quarter, two options. Book a demo and we will walk through our teleop rigs, annotation pipeline, and the instruction diversity QA gate live. Or download sample data: a free LeRobot-format sample pack of bimanual teleop episodes with segment-level language labels, so your team can inspect sync quality before you talk to anyone.

Request a Quote

Frequently Asked Questions

What is training data for vision-language-action (VLA) models?

It is a corpus of robot demonstration episodes pairing synchronized camera observations and robot actions with natural-language instructions. VLA models train on it, usually via imitation learning on top of a pretrained vision-language backbone, to map images plus an instruction to robot actions.

Pretraining-scale efforts use hundreds of thousands of episodes (OpenVLA used ~970k from Open X-Embodiment; pi-zero reports on the order of 10,000 hours). Fine-tuning an existing VLA to a new embodiment or task set typically needs far less: the literature shows meaningful results from 50-150 demonstrations per task on top of a broad base, with 500-2,000 hours being a common commercial fine-tuning corpus.

In DexSet’s benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity, language annotation adds $8-15 per data-hour, and QA adds $5-9. A 2,000-hour fully annotated corpus typically lands between $110k and $170k.

Not for most pretraining, since mainstream VLA backbones consume mono RGB. Stereo or depth earns its cost on contact-rich, cluttered, or precision tasks in your fine-tuning set, where mono depth ambiguity is a leading failure mode.

Only partially. Human egocentric video has no robot action labels, so it serves as a pretraining or co-training signal (the GR00T N1 data pyramid approach), improving visual and behavioral priors. The action-labeled teleoperation layer remains mandatory for the policy to output executable control.

Tokenized VLAs (RT-2, OpenVLA) discretize each action dimension into bins and predict actions as tokens, which suits language-model backbones and moderate control rates. Continuous-action VLAs (pi-zero with flow matching, Octo with diffusion) regress action chunks directly, enabling 50 Hz dexterous control but demanding tighter timestamp sync in the training data.