Teleoperation Archives

Case Study: How We Scaled VLA Training Data for a Humanoid Foundation Model Team

Posted on August 5, 2026July 30, 2026 by Enoch Pakanati

On station four of the collection floor, an operator sets down her leader arms after a clean bowl-to-sink transfer, and the batch dashboard flags the episode anyway. The instruction field reads “put the bowl in the sink,” the same sentence shape as two hundred episodes before it, and the template-share gate has just tripped. That flag, firing in the middle of a live program, is the single most important control in the engagement this post describes.

The client behind that program, a humanoid foundation model team, did not come to us asking for data. They came asking why their model was failing. Their OpenVLA-style fine-tune scored well on standard tabletop evaluations, then collapsed on the bimanual kitchen tasks their product actually needed. Worse, the policy only responded reliably to instructions phrased exactly like the training labels. “Put the bowl in the sink” worked. “Move that bowl into the sink” mostly did not.

The failure had a clean explanation in their data, and it is the most common one we see. Their existing corpus was large by episode count, roughly three times what we ended up collecting, but it covered only 9 task families and used 14 instruction templates. The model had learned those 9 tasks and those 14 sentence shapes, thoroughly, and almost nothing else. Volume was never the problem. Distribution was.

This post walks through what we scoped, what the QA gates caught, what it cost in structure (with our standard benchmark rates), and what changed in their evaluations. If you are planning a fine-tuning collection, the transferable part is the method: diversity targets set as hard gates, not aspirations.

Key Takeaways – 1,400 hours of bimanual teleoperation across 42 task families replaced a corpus 3x larger in episode count but 5x narrower in tasks. – A hard instruction diversity gate (no template above 8% of any batch) fixed paraphrase brittleness; language annotation ran at our standard $12-15 per data-hour, segment-level. – Blending new data 60/40 with the legacy corpus roughly doubled held-out kitchen task success in the client’s own evaluation harness. – QA rejected 11% of collected episodes; the top three causes were teleoperator mid-chunk pauses, frame-action sync drift, and failed-but-unlabeled episodes. – Total program ran 19 weeks from scoping to final delivery in LeRobot-compatible format.

The Starting Point: A Large but Narrow Corpus

A narrow corpus is a dataset whose episode count outruns its task, scene, and language diversity, and it trains models that look strong on in-distribution tests and brittle everywhere else. The client’s legacy data was a textbook case: high volume, 9 task families, 14 templates, single kitchen scene, one camera geometry.

The published evidence already pointed to the fix. Open X-Embodiment showed pooled diversity beating single-domain volume (arxiv.org/abs/2310.08864), and Mobile ALOHA reached strong per-task results co-training with roughly 50 demonstrations per task over a broad base (arxiv.org/abs/2401.02117). Our job was to apply that at commercial fine-tuning scale on their embodiment, where cross-embodiment transfer is weakest.

What We Scoped

The collection specification is the deliverable that matters most, so we fixed it before a single episode was recorded:

Volume and spread: 1,400 robot-hours of bimanual teleoperation across 42 task families, capped at 60 hours per family to force breadth.
Scenes and objects: 6 kitchen layouts, 300+ object instances, mandated lighting variation across sessions.
Capture spec: 4 camera streams per station (two exocentric, head, wrist), 30 Hz, hardware-timestamped, frame-to-action skew held under 10 ms.
Language spec: natural free-form instructions written at collection time, segment-level labels with timestamps on all long-horizon episodes, and the diversity gate below.
Format: LeRobot-compatible episodes so their training stack ingested data on day one (github.com/huggingface/lerobot).

The Instruction Diversity Gate

An instruction diversity gate is a batch-level QA rule that rejects any 1,000-episode batch where a single instruction template exceeds a set share, in this case 8%. Annotators drift toward comfortable phrasings; without a gate, “pick up the X and place it in the Y” quietly becomes half your labels.

We enforced three thresholds per batch: template share under 8%, minimum unique verb count, and minimum spatial-relation phrase coverage (“behind the cutting board”, “second shelf from the top”). Batches that failed went back for paraphrase augmentation before delivery. This is also where the money went: segment-level annotation with paraphrase passes ran at our standard $12-15 per data-hour, versus $8-10 for a plain episode-level pass. The client initially pushed back on that line item. It ended up being the line item their evaluation results vindicated most directly.

How the Program Ran

A collection program’s timeline is set by station-hours and reset speed, not by calendar ambition, so we planned backwards from throughput. The program ran 19 weeks across six bimanual stations: two weeks of scoping and task-family design with the client’s ML team, one week of operator training and pilot batches, fourteen weeks of production collection, and two weeks of final QA, re-collection of rejected hours, and packaging.

The pilot week matters more than its size suggests. We shipped the first 40 hours to the client immediately and they ran a smoke-test fine-tune before full production started. That surfaced two spec problems while they were still cheap to fix: their training stack expected a different gripper action convention, and one camera’s mounting height put the left workspace edge out of frame on tall-object tasks. Catching either in week twelve instead of week three would have cost hundreds of re-collected hours. If a vendor does not offer a pilot-batch gate before full-rate production, build one into the contract yourself.

Weekly deliveries went out in LeRobot format with per-batch QA reports, so the client’s dataset grew continuously instead of arriving as one end-of-program surprise.

What QA Caught

Episode rejection is the health metric of a collection program, and ours ran at 11% across the engagement. Where those episodes died:

Rejection Cause	Share of Rejects	Why It Matters for VLA Training
Teleoperator mid-chunk pauses	34%	Corrupts action-chunk targets for chunked policies (ACT-style and continuous heads)
Frame-action sync drift > 10 ms	27%	Measurably degrades continuous-action fine-tunes
Failed episode, unlabeled	22%	Success mislabeling flattens gains on precision tasks
Dropped frames / camera faults	11%	Breaks observation continuity
Instruction-episode mismatch	6%	Directly poisons language conditioning

Every rejected hour was recollected within the fixed budget because our per-hour rates ($40-60 for bimanual leader-follower work, our standard benchmark range) price in an expected rejection band. If a vendor quotes you a rate with no stated rejection policy, that risk is sitting on your side of the table.

Results

The client fine-tuned on the new corpus blended 60/40 with their legacy data and ran their internal evaluation harness before and after. Directional results, their harness, not an audited benchmark:

Held-out kitchen task success roughly doubled against the legacy-data baseline.
Paraphrased-instruction success moved from near-random to parity with template phrasing.
Long-horizon task completion improved most on tasks with segment-level labels, consistent with what the pi-zero pipeline reports about post-training data quality (arxiv.org/abs/2410.24164).

The result we quote most often is the second one, because it isolates the language layer. Nothing about the model changed. The labels changed.

What Transfers to Your Program

Four rules from this engagement apply to almost any VLA fine-tuning collection:

Audit distribution before buying volume; count task families and unique templates in your existing corpus first.
Set diversity thresholds as batch-level rejection gates, not guidelines.
Price annotation and QA as separate line items so you can see what you are cutting when you cut.
Cap hours per task family; the marginal episode of a covered task is your most overpriced purchase.

The full scoping framework, cost tables, and the RFP scorecard we build these programs from are in the complete guide to VLA training data.

Next Step

if your evaluations look like this client’s before-picture, book a scoping call. We will audit your corpus distribution before recommending a single collection hour, or start with the complete VLA training data guide.

Frequently Asked Questions

How many hours of data does it take to fine-tune a VLA for a humanoid?

This program used 1,400 hours across 42 task families, blended with existing data. Humanoids sit at the expensive end because cross-embodiment transfer from open datasets is weakest for them; standard single arms often need only 200-500 hours.

What is an instruction diversity gate?

A batch-level QA rule that rejects annotation batches where any single instruction template exceeds a set share (we used 8%), with minimum thresholds for unique verbs and spatial-relation phrases. It prevents the label monoculture that makes VLA policies brittle to paraphrasing.

Why blend new data with a legacy corpus instead of replacing it?

The legacy data still carried valid task knowledge; the 60/40 blend kept that signal while the new data corrected the distribution. Full replacement wastes paid-for coverage and can regress previously working tasks.

What rejection rate is normal in teleoperation collection?

We ran 11% on this program, and 10-15% is the band we consider healthy for bimanual long-horizon work. A vendor reporting near-zero rejection is usually not checking sync, pauses, or success labels.

Comparing Training Data Approaches for Vision-Language-Action (VLA) Models: Pros, Cons & Costs

Posted on August 4, 2026July 30, 2026 by Sainath Gupta

Zero robot action labels exist in the world’s largest manipulation video corpora. Thousands of hours of humans pouring, folding, and assembling, on YouTube and in Ego4D-scale datasets, and not one recorded motor command among them. That single number is why the VLA data sourcing debate never resolves cleanly: the cheapest, most scalable source of manipulation data is missing the one field an action model trains on, and every source that has the field costs real money per hour.

So the budget conversation inside most teams collapses into familiar camps. Someone wants to teleoperate everything because actions are the labels. Someone wants to generate everything in simulation because it is cheap. Someone read the GR00T paper and wants to pretrain on YouTube. All three are right about the strengths and quiet about the weaknesses, and the budget only covers one of them properly.

Our thesis, argued below with published results and our own collection benchmarks: these approaches are not substitutes, they are layers, with different label completeness, different transfer properties, and per-hour costs that differ by two orders of magnitude. Treating them as interchangeable is how teams end up with 40,000 synthetic episodes that fine-tune into a policy that cannot close a real gripper on a real deformable object.

This post compares the four practical sourcing approaches: teleoperation, human video, synthetic generation, and open datasets. For each you get what it is, where it wins, where it fails, and what it costs, ending with the decision matrix we use when scoping client collections. Costs marked as DexSet benchmarks are from our own production pipeline; the rest cite primary sources.

Key Takeaways – Teleoperation is the only approach that produces complete observation-action-language triples; it costs $28-60 per robot-hour in our benchmarks. – Human video is the cheapest per hour and the most scalable, but has no action labels; it works as a pretraining layer, never as the whole diet. – Synthetic data has near-zero marginal cost and perfect labels, but pays a sim-to-real tax on contact-rich tasks. – Open datasets (Open X-Embodiment, DROID, BridgeData V2) are effectively free and belong in every mixture; their limit is embodiment and domain mismatch. – The winning pattern in published systems is a pyramid: open + web data at the base, synthetic in the middle, embodiment-matched teleoperation on top.

Approach 1: Teleoperation

Teleoperation data is collected by a human operator directly controlling the robot while every observation and command is recorded, making the robot’s own action stream the training label. This is the approach behind ALOHA/ACT (arxiv.org/abs/2304.13705), DROID (arxiv.org/abs/2403.12945), and the top layer of essentially every serious VLA pipeline including pi-zero’s roughly 10,000-hour corpus (arxiv.org/abs/2410.24164).

Pros: complete and exact action labels; embodiment-matched by construction; supports contact-rich and bimanual tasks; language can be attached at collection time while context is fresh.

Cons: real-time throughput ceiling (20-40 usable tabletop episodes per hour on our rigs, 8-15 for long-horizon work); operator skill variance shows up in the data; rigs cost real money (an ALOHA-class bimanual station is around $20k before cameras).

Cost (DexSet benchmark): $28-38 per robot-hour for VR-rig tabletop collection, $40-60 for leader-follower bimanual work, plus $8-15 per data-hour for language annotation and $5-9 for QA.

Approach 2: Human Egocentric Video

Human video data is footage of people performing tasks, captured from head-mounted or fixed cameras, containing rich visual and behavioral signal but no robot action labels. Ego4D-scale corpora and in-house egocentric capture both fall here.

Pros: an order of magnitude cheaper per hour than teleoperation; humans perform tasks at natural speed with natural strategies; enormous diversity is reachable; head-mounted capture rigs are cheap to scale across many wearers.

Cons: no action labels, so it cannot train the action head directly; embodiment gap between human hands and robot grippers; requires retargeting or representation-learning machinery to become useful.

Cost (DexSet benchmark): $15-25 per hour for structured egocentric capture with task scripts and QA, roughly half the cost of teleoperation for the same wall-clock hour.

Where it fits: as pretraining and co-training signal. NVIDIA’s GR00T N1 explicitly structures this as the wide base of its data pyramid (arxiv.org/abs/2503.14734).

Approach 3: Synthetic and Simulation Data

Synthetic data is generated in physics simulators or by generative models, with actions and annotations produced programmatically at near-zero marginal cost. Perfect labels, infinite resets, no hardware wear.

Pros: marginal cost approaches zero after environment setup; perfect ground truth including states no sensor can capture; safe exploration of failure states; trivially parallelizable.

Cons: the sim-to-real gap is worst exactly where manipulation is hardest: contact dynamics, deformables, friction, specular and transparent objects. Environment engineering is a real up-front cost that budget lines often forget. Policies trained purely in sim routinely need real-data fine-tuning anyway.

Cost: environment and asset engineering is the dominant term (weeks of engineer time per task family); generation itself is compute-priced. Treat it as high fixed cost, near-zero marginal cost.

Approach 4: Open Datasets

Open datasets are published robot demonstration corpora, led by Open X-Embodiment with over one million trajectories across 22 embodiments (arxiv.org/abs/2310.08864), DROID with 76k episodes across 564 scenes, and BridgeData V2 with roughly 60k trajectories (arxiv.org/abs/2308.12952).

Pros: free; instantly available; proven at scale (OpenVLA pretrained on ~970k OXE episodes, arxiv.org/abs/2406.09246); increasingly standardized via LeRobot formats.

Cons: your embodiment, cameras, and tasks are probably not in there; instruction quality varies wildly between constituent datasets; license terms need actual reading; everyone else trains on the same data, so it confers no advantage by itself.

Side-by-Side Comparison

Dimension	Teleoperation	Human Video	Synthetic	Open Datasets
Action labels	Complete, exact	None	Perfect, programmatic	Complete, variable quality
Cost per data-hour	$28-60 + annotation (DexSet benchmark)	$15-25 (DexSet benchmark)	Near-zero marginal, high fixed	Free
Embodiment match	Exact	Poor (human)	Configurable	Rarely exact
Contact-rich fidelity	High	Visual only	Weak to moderate	High where present
Scale ceiling	Low (real time)	High	Very high	Fixed (~1M episodes)
Language quality	Controllable at source	Must be added	Templated by default	Inconsistent
Best role	Fine-tuning top layer	Pretraining base	Mid-layer augmentation	Pretraining base

The Decision Matrix We Actually Use

The right mixture follows from two questions: how far is your embodiment from the open corpora, and how contact-rich are your target tasks?

Standard single arm, tabletop tasks: open datasets + a thin teleop fine-tuning layer (200-500 hours). Cheapest viable path.
Standard arm, contact-rich or precision tasks: add stereo teleop collection on the target tasks; synthetic helps for coverage but will not close the contact gap alone.
Dexterous hands or humanoids: cross-embodiment transfer is weakest here. Budget for substantial embodiment-matched teleoperation (1,000+ hours) over a human-video and open-data base.
Long-horizon mobile manipulation: prioritize segment-level language labels; Mobile ALOHA showed strong results co-training with roughly 50 demos per task on a broad base (arxiv.org/abs/2401.02117), so spread hours across many tasks.

Two budgeting notes that surprise first-time buyers. First, annotation and QA scale with data-hours, not with approach, so a cheap collection method with poor labels often costs more per useful training hour than an expensive one delivered clean; a $30 teleop hour that ships with diverse language labels and verified success flags beats a $15 hour that needs a relabeling pass and a 20% cull. Second, the approaches interact: a few hundred hours of well-instrumented teleoperation is also the calibration set that makes your synthetic pipeline honest, because it gives you real contact dynamics to validate against. Teams that sequence it that way (teleop first, sim scaled second) report far less sim-to-real disappointment than teams that generate first and verify never.

The pattern across every row: nobody should pay collection rates for the base of the pyramid, and nobody should expect the free layers to cover the top. For the full scoping framework, mixture ratios, and our vendor RFP scorecard, see the complete guide to VLA training data.

Next Step

if you are weighing these approaches for a real budget, read the complete VLA training data guide or book a demo and we will pressure-test your planned mixture against our collection benchmarks.

Frequently Asked Questions

What is the cheapest way to get training data for a VLA model?

Start with free open datasets (Open X-Embodiment, DROID, BridgeData V2) for pretraining, then buy only embodiment-matched teleoperation for fine-tuning. In our benchmarks that fine-tuning layer costs $28-60 per robot-hour plus annotation, and 200-500 hours is often enough for standard manipulators.

Is synthetic data good enough to train a VLA model?

Not alone. Synthetic data provides cheap coverage and perfect labels but carries a sim-to-real gap that is worst on contact dynamics and deformable objects. Published systems use it as a middle layer between web-scale video and real teleoperation.

How does teleoperation data compare to human video on cost?

In DexSet’s pipeline, structured egocentric human video runs $15-25 per hour versus $28-60 per robot-hour for teleoperation. The difference is labels: teleoperation records executable robot actions, human video does not, so they occupy different layers of the training mixture rather than competing.

Should I collect my own data or use Open X-Embodiment?

Both. OXE-scale open data is the standard pretraining base and costs nothing. Custom collection earns its price where open data is thin: your specific embodiment, your camera geometry, your tasks, and instruction phrasing diverse enough to make language conditioning work.

Why Training Data for Vision-Language-Action (VLA) Models Is the Biggest Bottleneck in Physical AI

Posted on August 3, 2026July 30, 2026 by Enoch Pakanati

Somewhere on every VLA data plan sits a line that reads “annotation and QA,” and it is the checklist item nearly everyone skips when the budget gets real. Collection hours get scoped to the dollar. GPU spend gets scoped to the node. The layers that decide whether those hours train anything get a checkbox. That skipped line is usually where we find the explanation when a team arrives with a Prismatic-style backbone, sensible action head, clean training code, and a model that still cannot pick up a mug it has never seen when the instruction is phrased slightly differently.

The deeper reason is structural, not a bug. Language models got to scrape a web that humanity spent thirty years writing. VLA models need trajectories where a physical robot moved and every action was recorded, and that data did not exist until someone paid to create it, one teleoperated episode at a time. Open X-Embodiment, the largest pooled open corpus, holds just over one million trajectories from 22 embodiments (arxiv.org/abs/2310.08864). Web-scale image-text corpora hold billions of pairs. Physical AI is trying to reach foundation-model behavior on roughly one-thousandth of the data its sibling fields consider table stakes.

This post argues one thesis: the VLA data bottleneck is not a single constraint but a four-layer stack (collection throughput, action-label fidelity, language annotation, QA), and teams that budget only for the first layer buy datasets that train worse models than their size suggests. We show where each layer binds, what it costs to push against, and which strategies the leading labs use to spend around it efficiently. We collect and QA this data for a living at DexSet, so the cost numbers are our own production benchmarks, flagged as such.

Key Takeaways – The robot data gap is around three orders of magnitude versus web vision-language corpora, and it cannot be closed by scraping. – The bottleneck has four layers: collection throughput, action-label fidelity, language annotation quality, and QA. Most teams only budget for the first. – RT-1 took 17 months and 13 robots to produce ~130k episodes; that throughput ceiling is why teleoperation economics matter. – In our benchmarks, usable teleop data costs $28-60 per robot-hour to collect, plus $8-15 per hour for language annotation and $5-9 for QA. – The efficient escape routes are cross-embodiment pretraining, web co-training, and spending fresh budget on diversity rather than raw episode count.

What Makes VLA Training Data a Bottleneck?

The VLA data bottleneck is the mismatch between how fast robot demonstration data can be produced and how much of it foundation-model training consumes. Every VLA needs episodes containing synchronized observations, executed actions, and language instructions, and each of those three elements has its own production constraint.

Consider the throughput math. RT-1’s dataset took a fleet of 13 robots 17 months to reach roughly 130k episodes across about 700 tasks (arxiv.org/abs/2212.06817). That is real-world, well-funded, industrial-scale collection, and it averages out to under 20 usable episodes per robot per day once resets, failures, and curation are counted. On our own rigs we see 20-40 usable tabletop episodes per collection hour on a good day, and 8-15 for long-horizon bimanual work. There is no caching layer for the physical world. Every episode costs wall-clock time on hardware.

The Four Layers of the Bottleneck

The bottleneck is a stack of four constraints, and underestimating any one of them produces a dataset that trains a worse model than its size suggests.

1. Collection throughput. Robots and teleoperators produce data in real time. A 10,000-hour corpus, the scale pi-zero reports pretraining on (arxiv.org/abs/2410.24164), is five years of single-rig operation or a coordinated fleet effort. Throughput is bought with parallel stations, trained operators, and fast reset design, all of which show up in cost per hour.

2. Action-label fidelity. Actions are the labels, and they are only as good as your sync. Continuous-action models (flow matching, diffusion heads) train on action chunks, and timestamp skew between frames and commands degrades them measurably. We hold frame-to-action skew under 10 ms on our rigs because our fine-tune ablations show loss curves separating beyond that.

3. Language annotation quality. An episode without a good instruction is imitation data, not VLA data. Templated labels teach a dozen verbs; natural, diverse instructions teach language conditioning. This layer adds $8-15 per data-hour in our pipeline, and skipping it is the single most common false economy we see.

4. QA and success verification. Failed episodes, teleoperator pauses, dropped frames, and mislabeled successes all pollute training. A 5-10% contamination rate is enough to flatten fine-tune gains on precision tasks, which is why QA is a priced line item and not a virtue.

Why You Cannot Scrape Your Way Out

Web video contains no action labels, so it cannot substitute for the action-labeled core of VLA training. Human video (Ego4D-scale corpora, YouTube) shows what manipulation looks like but never records the motor commands that produced it. The field’s answer, formalized in NVIDIA’s GR00T N1 data pyramid (arxiv.org/abs/2503.14734), is to use web and human video as a broad lower layer for visual priors, synthetic data in the middle, and real teleoperation as the small, expensive, indispensable top layer. The pyramid reduces how much teleop you need. It does not remove the layer.

What the Bottleneck Costs, in Numbers

The cost of VLA training data is best expressed per robot-hour with annotation and QA broken out, because bundled per-episode pricing hides where the money goes. Our production benchmarks:

Layer	Typical Cost	Failure Mode If Skipped
Teleop collection (tabletop, VR rig)	$28-38 per robot-hour	None; this is the floor
Teleop collection (bimanual, leader-follower)	$40-60 per robot-hour	Under-scoped dexterity data
Language annotation (episode-level)	$8-10 per data-hour	Policy ignores instructions
Language annotation (segment-level)	$12-15 per data-hour	No long-horizon capability
QA + success labels	$5-9 per data-hour	5-10% contamination, flattened gains

A 2,000-hour fine-tuning corpus, fully annotated and QA’d, lands between roughly $110k and $170k. That number stops feeling large when compared against the GPU spend it protects: a fine-tune run on contaminated or label-poor data burns the same compute and returns a weaker policy.

How Leading Teams Route Around It

The efficient strategies all share one principle: never pay to collect what transfer can give you, and spend fresh budget on what transfer cannot.

Start from cross-embodiment pretraining. OpenVLA’s 7B model, pretrained on ~970k OXE episodes, outperformed the 55B RT-2-X on its 29-task evaluation (arxiv.org/abs/2406.09246). The base layer is a solved purchase; open weights exist.
Co-train with web vision-language data. RT-2 showed this preserves open-vocabulary semantics (arxiv.org/abs/2307.15818). It costs mixture engineering, not collection dollars.
Buy diversity, not depth. Past roughly 50-150 demonstrations per task, marginal episodes decay in value; Mobile ALOHA reached strong task performance co-training with about 50 demos per task (arxiv.org/abs/2401.02117). New tasks, scenes, objects, and phrasings beat episode 400 of a solved task.
Match embodiment where it counts. Cross-embodiment transfer is weakest into dexterous hands and humanoids, so reserve embodiment-matched collection for exactly those gaps.

One more route deserves a mention because it is quietly effective: instruction relabeling. DROID and several lab pipelines have used vision-language models to generate or augment instructions on already-collected episodes, which raises language diversity on data you have already paid for. It does not fix bad sync or unlabeled failures, and machine-generated labels need a human QA sample, but as a cost lever it is hard to beat: you are improving the most neglected layer of the stack at annotation prices rather than collection prices.

The common thread in all five strategies is that they treat the bottleneck as a portfolio problem. No single source escapes it. A deliberate stack of open data, web co-training, synthetic coverage, relabeling, and targeted teleoperation gets a team to a working policy at a fraction of the naive all-teleop cost, and every published frontier system is some version of that stack.

We cover the full scoping framework, including the RFP scorecard we make vendors fill out, in our complete guide to VLA training data.

Next Step

if you want to see what clean, sync-verified, language-diverse teleop data looks like before budgeting a collection, download our free LeRobot-format sample pack, or read the full guide first.

Frequently Asked Questions

Why is training data the bottleneck for VLA models instead of compute?

Compute is fungible and rentable; robot demonstrations are produced in real time on physical hardware. The largest open robot corpus holds about one million trajectories, roughly one-thousandth of web vision-language scale, and every additional hour must be physically collected.

How do I scale training data for VLA models on a startup budget?

Pretrain or start from open cross-embodiment checkpoints (OpenVLA, Octo), co-train with web data to keep semantics, and spend collection budget only on embodiment-matched, task-matched fine-tuning data with diverse language labels. Expect $110k-170k for a fully annotated 2,000-hour corpus at current market rates.

Can simulation solve the VLA data bottleneck?

Simulation reduces but does not remove real-data needs. Sim-to-real gaps in contact dynamics and visual realism persist, which is why current pipelines (including GR00T N1) use synthetic data as a middle layer under a real teleoperation layer, not as a replacement.

How many demonstrations per task does a VLA fine-tune need?

The imitation-learning literature repeatedly lands around 50-150 demonstrations per task when fine-tuning on top of a broad pretrained base, with more needed for contact-rich or long-horizon tasks. Diversity across tasks and phrasings matters more than pushing any one task far past that floor.

The Complete Guide to Training Data for Vision-Language-Action (VLA) Models (2026)

Posted on July 31, 2026July 30, 2026 by Sainath Gupta

Five billion image-text pairs sit in LAION-5B (arxiv.org/abs/2210.08402); just over one million robot trajectories sit in Open X-Embodiment, the largest open robot dataset ever assembled (arxiv.org/abs/2310.08864). Those two numbers should not both describe the training diet of a single model family, yet every production VLA pairs a vision-language half fed at web scale with an action half starved at roughly one-thousandth of it. That mismatch, more than any architecture choice, is the defining constraint of physical AI in 2026.

The gap exists because the two halves come from different worlds. The vision-language half was pretrained on image-text pairs that already existed. The action half needs robot trajectories that do not exist until someone physically collects them, frame-synchronized, action-labeled, and paired with language instructions. You cannot scrape your way out of that. Which is why, on most VLA programs, the architecture question gets settled in a week and the compute budget in a month, while the question of how many language-annotated trajectories the model needs, at what task diversity, on which embodiments, goes unanswered the longest.

This guide gives you the full picture: what VLA training data is, how the landmark models (RT-1, RT-2, OpenVLA, Octo, pi-zero, GR00T) actually sourced their data, how action representation changes your labeling requirements, what collection and annotation cost per hour, and how to scope a dataset that transfers to your embodiment instead of overfitting to someone else’s lab. The thesis we argue throughout, and the one our delivery data keeps confirming: once you stand on the open-data base layer, dataset composition (task diversity, instruction diversity, capture fidelity) predicts VLA capability better than episode count, so budget should follow distribution, not volume.

DexSet supplies egocentric, exocentric, teleoperation, mono, and stereo data to robot foundation model teams. The numbers in this guide come from our own rigs, QA pipelines, and cost benchmarks, alongside the primary papers. Where a figure is our internal benchmark rather than a published result, we say so.

TL;DR: Key Takeaways – VLA training data consists of robot trajectories (synchronized camera frames + robot actions) paired with natural-language instructions, usually collected via teleoperation. – Landmark scale points: RT-1 used ~130k episodes over 17 months; OpenVLA pretrained on ~970k episodes from Open X-Embodiment; pi-zero reports on the order of 10,000 hours of dexterous robot data. – Action representation drives labeling spec: discrete action tokens (RT-2, OpenVLA) tolerate lower control rates; continuous-action heads (pi-zero flow matching, Octo diffusion) reward 50 Hz, low-jitter capture. – In our benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity; a proper language annotation pass adds $8-15 per hour of data. – Data mixture beats raw volume: co-training on web-scale vision-language data plus a well-balanced robot mixture is what gives VLAs semantic generalization. – Task and scene diversity predict generalization better than episode count once you pass a per-task floor (roughly 50-150 demonstrations per task in the imitation-learning literature).

What Is Training Data for Vision-Language-Action (VLA) Models?

Training data for VLA models is a corpus of robot demonstration episodes in which each timestep pairs visual observations, robot proprioceptive state, and executed actions, and each episode (or segment) carries a natural-language instruction describing the task. A VLA model consumes images and an instruction as input and produces robot actions as output, so its training data must contain all three aligned in time.

A single well-formed episode contains:

Observations. One or more camera streams (wrist-mounted, over-the-shoulder exocentric, or head-mounted egocentric), typically 224×224 to 1080p RGB at 10-60 fps, optionally stereo pairs or depth.
Proprioception. Joint positions, gripper state, end-effector pose, usually at the control rate.
Actions. The command stream that produced the motion: delta end-effector poses, joint targets, or gripper commands, timestamped against frames.
Language. An instruction such as “put the red mug on the top shelf,” attached at episode level, and increasingly at sub-episode segment level for long-horizon tasks.

The entity chain matters for anyone mapping this space: teleoperation rigs (ALOHA-style leader-follower arms, VR controllers, exoskeletons) produce demonstrations; demonstrations feed imitation learning; imitation learning at scale, on top of a pretrained vision-language backbone, is what turns a VLM into a VLA. The Hugging Face LeRobot project has become the de facto open format for storing these episodes, and its dataset cards are a good reference for what a clean schema looks like.

How the Landmark VLA Models Sourced Their Data

Every major VLA result is downstream of a specific, documented data strategy, and the strategies differ more than the architectures do. Reading the data sections of these papers teaches you more about building a VLA than reading the model sections.

Model (Year)	Org	Params	Robot Data	Language Source	Action Representation
RT-1 (2022)	Google	35M	~130k episodes, 13 robots, 17 months, ~700 tasks	Templated instructions	Discrete tokens, 256 bins (arxiv.org/abs/2212.06817)
RT-2 (2023)	Google DeepMind	12B / 55B	RT-1 data, co-fine-tuned with web-scale VLM data	Templated + web text	Actions as text tokens (arxiv.org/abs/2307.15818)
Octo (2024)	UC Berkeley et al.	27M / 93M	800k trajectories from Open X-Embodiment	Mixed, dataset-dependent	Diffusion head, continuous (arxiv.org/abs/2405.12213)
OpenVLA (2024)	Stanford et al.	7B	970k episodes from Open X-Embodiment	Mixed OXE instructions	Discrete tokens (arxiv.org/abs/2406.09246)
pi-zero (2024)	Physical Intelligence	~3B	On the order of 10k hours, 7 platforms, 68 task families	Natural + segment-level relabeling	Flow matching, 50 Hz continuous (arxiv.org/abs/2410.24164)
GR00T N1 (2025)	NVIDIA	2B	Data pyramid: web video + synthetic + real robot data	Mixed human + generated	Continuous, diffusion-based (arxiv.org/abs/2503.14734)

Three lessons sit in that table. First, scale climbed from 130k episodes to roughly a million in two years, but the biggest capability jumps (RT-2 over RT-1, pi-zero over Octo-class models) came from mixture changes, not just volume. Second, OpenVLA at 7B outperformed the 55B RT-2-X on the paper’s 29-task evaluation suite, which tells you curated cross-embodiment data can substitute for an order of magnitude of parameters. Third, nobody serious trains on robot data alone anymore; co-training against web-scale vision-language corpora is standard because it is what keeps semantic knowledge (what “the leftmost ripe banana” means) alive while the model learns motor control.

Core Data Modalities for VLA Training

Modality choice determines what your VLA can learn and what your collection rig must capture. The main axes are viewpoint (egocentric vs exocentric), depth strategy (mono vs stereo), and collection method (teleoperation vs human video).

Egocentric vs Exocentric Views

Egocentric data is captured from the robot’s or demonstrator’s own viewpoint; exocentric data is captured from external cameras watching the workspace. Wrist and head cameras give the policy view-invariant, occlusion-realistic inputs and tend to transfer better across scene layouts. Exocentric views give annotators and QA reviewers the context to verify task success and label segments. In our pipelines we treat exocentric streams as partly a QA instrument: an episode with only a wrist view is roughly twice as expensive to verify because reviewers must scrub proprioception logs to confirm success.

Mono vs Stereo

Mono capture records a single RGB stream per viewpoint; stereo capture records calibrated pairs that support depth estimation. Most published VLAs train on mono RGB because the backbone VLMs are mono. Stereo earns its extra cost on contact-rich and clutter-heavy manipulation, where depth ambiguity is a dominant failure mode. Our rule of thumb: mono for pretraining breadth, stereo for the fine-tuning set on your target tasks.

Teleoperation vs Human Video

Teleoperation produces the gold standard because the action stream is the robot’s own. Leader-follower rigs like ALOHA (arxiv.org/abs/2304.13705) hit high dexterity at a hardware cost around $20k per station; VR-based rigs are cheaper and faster to train operators on but add pose-retargeting noise. Human egocentric video (Ego4D, EgoExo4D scale) has no action labels, so it enters VLA training as a co-training or pretraining signal, the approach GR00T N1 formalizes as a data pyramid. Human video buys you visual and behavioral priors cheaply; it never replaces the teleop layer where actions live.

Action Representation: Tokenization vs Continuous Actions

Action representation is the choice of how robot actions are encoded for the model to predict, and it silently rewrites your data spec. There are two dominant families.

Discrete action tokenization (RT-1, RT-2, OpenVLA) bins each action dimension, typically into 256 buckets, and treats action prediction as next-token prediction. It plugs directly into a language-model backbone and tolerates modest control rates. The FAST tokenizer (arxiv.org/abs/2501.09747) pushed this further with a compression-based scheme that makes autoregressive VLAs viable on high-frequency data.

Continuous action heads (Octo’s diffusion head, pi-zero’s flow matching) regress action chunks directly. They excel at dexterous, high-rate control (pi-zero runs 50 Hz) but they punish sloppy capture. Timestamp jitter between frames and actions that a tokenized model shrugs off will smear a flow-matching target distribution.

What this means for data buyers, from our QA logs:

For tokenized targets, 10-15 Hz aligned capture is usually sufficient; invest the savings in task diversity.
For continuous heads, capture at 30-60 Hz with hardware-timestamped sync; we hold frame-to-action skew under 10 ms on our teleop rigs because beyond that we can measure it in fine-tune loss.
Action-chunk prediction (introduced with ACT on ALOHA) means episodes must be free of teleoperator pauses mid-chunk, which is a QA check most teams add only after their first bad fine-tune.

Language Annotation: The Layer Everyone Underestimates

Language annotation is the process of attaching instructions to robot episodes, and it is where VLA datasets most often quietly fail. A VLA is only as language-conditioned as its labels are diverse.

Templated vs natural instructions. RT-1-era datasets used templates (“pick X”, “move X near Y”). Templates are cheap and consistent, and they teach the model a grammar of about a dozen verbs. Models trained on templated instructions follow paraphrases poorly; the semantic generalization RT-2 demonstrated came from the web co-training, not the robot labels. Current practice, visible in the pi-zero and DROID (arxiv.org/abs/2403.12945) pipelines, is natural free-form instructions plus paraphrase augmentation.

Episode-level vs segment-level labeling. Episode-level labeling attaches one instruction to a whole demonstration. Segment-level labeling breaks a long-horizon episode into sub-tasks (“open the drawer”, “place the spoon inside”, “close the drawer”) with aligned timestamps. Segment labels cost more but unlock long-horizon training and better credit assignment. In our pipeline, segment-level labeling costs roughly 2.5x the episode-level pass because annotators must scrub video and mark boundaries, not just watch and describe.

Our first-hand benchmark: a language annotation pass adds $8-15 per hour of robot data depending on instruction density and whether we run single or dual annotation. We also run an instruction diversity QA gate: across any 1,000-episode batch we require minimum thresholds for unique verbs, unique object references, and spatial-relation phrases, because we have watched fine-tunes on label-poor batches lose the ability to follow anything but the majority phrasing.

Data Mixtures, Co-Training, and Cross-Embodiment Transfer

A data mixture is the weighted combination of datasets a VLA trains on, and mixture design is now a first-class engineering discipline. Three decisions dominate.

Web co-training ratio. RT-2 established that co-fine-tuning on web vision-language data alongside robot data preserves semantic breadth. Teams that fine-tune on robot data alone consistently report the model “forgetting” open-vocabulary grounding.

Cross-embodiment weighting. Open X-Embodiment showed that pooling data across 22 robot types improves average success, and OpenVLA and Octo both hand-tuned per-dataset mixture weights rather than sampling uniformly. Transfer across embodiments is real but not free: it appears strongest between similar morphologies (single-arm manipulators with parallel grippers) and weakest into dexterous hands and humanoids, which is exactly why humanoid programs still commission embodiment-matched collection.

There is no published universal mixture ratio, and be suspicious of anyone who quotes one. What the papers do agree on is the shape of the curve: heavily downweight any single dataset that would dominate by timestep count, keep a meaningful web co-training fraction throughout fine-tuning rather than dropping it to zero, and upweight the small datasets that match your target embodiment and camera geometry. In practice we see teams land between 10% and 40% embodiment-matched data by timestep in their final fine-tuning mixtures, with the exact point set by held-out evaluation rather than by principle. Measure mixtures in timesteps or tokens, not episodes; episode lengths vary enough between sources that episode-level accounting quietly misstates composition by 2-3x.

Diversity vs depth. Once a task passes a demonstration floor (Mobile ALOHA showed strong results co-training with as few as 50 demonstrations per task on top of a broad base, arxiv.org/abs/2401.02117), the marginal episode of task 12 is worth less than the first episode of task 13. When we rebalance client collections, we usually shift budget from episode count toward scene, object, and instruction diversity, and the downstream evaluations reward it.

Cost and Economics: What VLA Training Data Actually Costs

VLA data cost is driven by four stacked line items: rig amortization, operator time, annotation, and QA. Vendors who quote a single opaque per-episode price are hiding this stack. Here is ours, from DexSet production benchmarks (round figures, typical ranges we see):

Cost Component	Typical Range	Notes
Teleop collection (VR rig, tabletop tasks)	$28-38 per robot-hour	Includes operator, station amortization, resets
Teleop collection (leader-follower, bimanual/contact-rich)	$40-60 per robot-hour	ALOHA-class rigs, slower resets, higher operator skill
Language annotation, episode-level	$8-10 per data-hour	Natural instructions, single pass
Language annotation, segment-level + paraphrases	$12-15 per data-hour	Timestamped sub-task boundaries, dual pass on 10%
QA and success verification	$5-9 per data-hour	Success labels, sync checks, instruction diversity gate
Useful episodes per collection hour	20-40 tabletop; 8-15 long-horizon	After discarding failed/aborted episodes

Worked example: a 2,000-hour bimanual manipulation corpus with segment-level language and full QA lands between $114k and $168k. Against that, note that pi-zero-class results were built on roughly 10,000 hours. This is why mixture design and cross-embodiment pretraining matter commercially: nobody fine-tuning a foundation model should be paying to recollect the base layer that OXE-scale open data already provides. Pay for the embodiment-matched, task-matched, well-labeled top of the pyramid.

Case Study Proof: Scaling a Fine-Tuning Corpus for a Humanoid Team

A humanoid foundation model team came to us with an OpenVLA-style base model and a fine-tuning problem: strong tabletop scores, weak performance on their bimanual kitchen tasks, and instructions that only worked when phrased like the training templates. We scoped a 1,400-hour teleoperation collection across 42 task families with a hard instruction-diversity requirement (no template allowed to exceed 8% of a batch) and segment-level labels on all long-horizon episodes.

The mixture, not the volume, did the work. Their prior corpus had 3x our episode count but 9 task families and 14 instruction templates. After fine-tuning on the new corpus blended 60/40 with their legacy data, their internal evaluation showed task success on held-out kitchen tasks roughly doubling, and paraphrased-instruction success moved from near-random to parity with template phrasing. We report those as directional results from the client’s own evaluation harness, not audited benchmarks. The full breakdown is in our case study write-up.

Scoping Your Own Collection: The RFP Scorecard

A VLA data RFP scorecard is a structured checklist for comparing data vendors on the dimensions that predict fine-tune quality. If you take one action from this guide, make vendors answer these in writing:

Frame-to-action sync tolerance, in milliseconds, with measurement method.
Control and capture rates, and whether timestamps are hardware or software.
Instruction sourcing: templated, natural, or paraphrase-augmented, with a diversity metric per batch.
Episode-level vs segment-level labeling, priced separately.
Success-labeling protocol and inter-annotator agreement.
Cost per robot-hour, broken into collection, annotation, and QA. Refuse bundled per-episode pricing.
Delivery format (LeRobot-compatible schemas save your team weeks).

We publish ours as a downloadable template. [Download the VLA Data RFP Scorecard] and use it against us too.

Related reading this week:

Next Step

If you are scoping a VLA data collection this quarter, two options. Book a demo and we will walk through our teleop rigs, annotation pipeline, and the instruction diversity QA gate live. Or download sample data: a free LeRobot-format sample pack of bimanual teleop episodes with segment-level language labels, so your team can inspect sync quality before you talk to anyone.

Frequently Asked Questions

What is training data for vision-language-action (VLA) models?

It is a corpus of robot demonstration episodes pairing synchronized camera observations and robot actions with natural-language instructions. VLA models train on it, usually via imitation learning on top of a pretrained vision-language backbone, to map images plus an instruction to robot actions.

How much robot data does a VLA model need?

Pretraining-scale efforts use hundreds of thousands of episodes (OpenVLA used ~970k from Open X-Embodiment; pi-zero reports on the order of 10,000 hours). Fine-tuning an existing VLA to a new embodiment or task set typically needs far less: the literature shows meaningful results from 50-150 demonstrations per task on top of a broad base, with 500-2,000 hours being a common commercial fine-tuning corpus.

What does VLA training data cost?

In DexSet’s benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity, language annotation adds $8-15 per data-hour, and QA adds $5-9. A 2,000-hour fully annotated corpus typically lands between $110k and $170k.

Do I need stereo or depth data for VLA training?

Not for most pretraining, since mainstream VLA backbones consume mono RGB. Stereo or depth earns its cost on contact-rich, cluttered, or precision tasks in your fine-tuning set, where mono depth ambiguity is a leading failure mode.

Can I train a VLA on human video instead of teleoperation?

Only partially. Human egocentric video has no robot action labels, so it serves as a pretraining or co-training signal (the GR00T N1 data pyramid approach), improving visual and behavioral priors. The action-labeled teleoperation layer remains mandatory for the policy to output executable control.

What is the difference between action tokenization and continuous actions?

Tokenized VLAs (RT-2, OpenVLA) discretize each action dimension into bins and predict actions as tokens, which suits language-model backbones and moderate control rates. Continuous-action VLAs (pi-zero with flow matching, Octo with diffusion) regress action chunks directly, enabling 50 Hz dexterous control but demanding tighter timestamp sync in the training data.

Case Study: How We Scaled Data Capture Hardware for Embodied AI for a VLA Model

Posted on July 29, 2026July 24, 2026 by Enoch Pakanati

Two numbers sat in the same planning deck: 40 captured hours a week, and 8,000 hours due in five months. Both were accurate, and they could not both survive. Hitting 8,000 hours in roughly 22 weeks means averaging about 370 delivered hours a week, nine times what the floor was producing; at the actual run rate, the dataset would have arrived roughly four years late, well after the funding milestone it was supposed to unblock. This is the point where a VLA foundation model team brought the problem to us.

The gap was not effort, and that is the thesis of this case study: scaling capture is an architecture problem (sync, formats, parallel cheap seats), not an effort or premium-hardware problem. Their two-person capture team worked hard on good hardware, a single ALOHA-class station and a Vision Pro teleop pilot. The gap was architecture: no sync strategy across devices, no shared episode format, and a fleet plan that scaled capex faster than throughput. This is the most common shape of the scaling problem we see, so we are publishing the anonymized playbook.

This post walks through the three phases of the engagement: the audit, the hardware re-spec, and the scale-out, with the real throughput and cost numbers at each stage. If you are sizing a capture floor for VLA fine-tuning, the tables below are the ones we wish every team had before their first purchase order.

DexSet operates egocentric, teleoperation, and exocentric capture lines in production; this engagement ran on our floors with the client’s task list and QA criteria. Per our client agreements, the team is anonymized (“a VLA foundation model team”) and all numbers are rounded operational figures.

Key Takeaways – A VLA foundation model team needed 8,000 hours of manipulation data in five months; their in-house rate was 40 hours a week. We reached a sustained 400 hours a week by week nine. – The fix was mostly architectural: hardware sync, one episode format (LeRobot-compatible HDF5), and parallel cheap seats instead of premium single rigs. – Fleet mix at full scale: 6 GELLO-equipped arm cells, 3 ALOHA-class bimanual stations, 12 Quest 3 teleop seats, 20 UMI-style grippers, and one 6-camera D455 evaluation studio. – Delivered cost settled at $31-42 per teleop hour across task families, inside our standard $28-60 benchmark range.

Phase 1: The Audit (Weeks 1-2)

A capture audit is a measured baseline of usable hours per seat per shift, and it always starts by separating captured hours from delivered hours. The client’s 40 weekly hours were captured hours; after QA, delivered hours were closer to 26. Two findings explained most of the loss.

First, synchronization. The Vision Pro pilot streamed hand pose to the robot over Wi-Fi with no shared clock, and QA was rejecting roughly a quarter of episodes for observation-action misalignment beyond their 25 ms tolerance. Second, format debt: the ALOHA station wrote its own HDF5 layout, the Vision Pro stack wrote JSON pose logs plus video, and an engineer spent about a day a week converting between them.

Neither problem is exotic. Both are invisible until you measure delivered hours instead of captured hours.

Phase 2: The Hardware Re-Spec (Weeks 3-5)

A hardware re-spec is a redesign of the fleet around delivered cost per hour rather than device quality, and ours cut the premium hardware first. The Vision Pro seats moved to evaluation-only duty. For fleet teleop we standardized on Quest 3 headsets at roughly $500 per seat: seven times cheaper per seat, no measurable difference in demonstrations per operator-hour on the client’s pick, place, and insertion task families, and cheap enough to hold 20 percent spares.

For arm-based capture we kept the existing ALOHA-class station (fine bimanual tasks justified it) and added GELLO leader arms (arxiv.org/abs/2309.13037) at about $300 per arm on follower robots the client already owned. For task families that did not need force or proprioception fidelity, we added UMI-style handheld grippers (arxiv.org/abs/2402.10329) at roughly $400 per unit, with a budgeted SLAM QA pass.

The last addition was a six-camera RealSense D455 evaluation studio, hardware-synced through the D455’s trigger pin, wrapped around one teleop cell. Every evaluation episode was therefore captured ego and exo simultaneously, which the client’s research team used to test viewpoint generalization in the style of Ego-Exo4D (arxiv.org/abs/2311.18259). The studio cost about $2,700 in cameras and roughly the same again in truss, sync wiring, lighting, and NVMe ingest, a ratio worth budgeting for in advance.

Every line wrote one format: LeRobot-compatible episodes (github.com/huggingface/lerobot) with per-frame timestamps from a shared NTP-disciplined clock and a hardware sync signal on the camera side. The conversion engineer got their day back.

Phase 3: Scale-Out (Weeks 6-9)

Scale-out is the replication of a proven seat design in parallel, and it only works after yield per seat is stable. We added seats in weekly increments, holding the QA gate constant, until the floor hit the target run rate.

Metric	Baseline (client in-house)	Week 5 (re-spec live)	Week 9 (full scale)
Captured hours / week	40	130	~460
Delivered hours / week (post-QA)	26	110	~400
QA rejection rate	~35%	15%	10-12%
Active seats	3	14	42
Fleet capex (cumulative)	~$32k	~$68k	~$118k
Delivered cost per teleop hour	not tracked	$46	$31-42

Two numbers deserve emphasis. The rejection rate fell from 35 percent to under 12 percent before most of the new seats arrived; sync and format fixes, not hardware volume, did that. And full-scale fleet capex, about $118k for 42 seats, is less than many teams spend on four premium bimanual cells. Cheap synchronized seats, bought after the yield problem was solved, are what made the 8,000-hour target arrive on schedule.

The non-hardware work mattered too, and it is easy to undercount. Operator onboarding became a two-day curriculum with a certification task instead of an afternoon of shadowing, which pulled new-seat ramp time down from three weeks to one. The QA gate ran four automated checks (sync residual, calibration freshness, trajectory smoothness, format validity) plus human review on a 10 percent sample, and its thresholds never moved during scale-out. Holding the gate constant is what made the weekly delivered-hours number trustworthy enough to plan a training schedule against.

What We Would Do Differently

Honest retrospectives are rarer than case studies, so here is ours. We would have moved the UMI-style gripper line earlier in the schedule; its SLAM post-processing pipeline took three weeks to tune, and starting it in week 3 instead of week 6 would have added several hundred delivered hours. We also underestimated storage: the exo studio’s terabyte-per-shift output forced a mid-project NVMe retrofit that a day of arithmetic would have predicted. Neither mistake threatened the deadline. Both are now checklist items in every new engagement.

What Generalizes From This Engagement

The transferable lessons are hardware-selection rules, and they match the decision matrix in our full guide to data capture hardware for embodied AI:

Measure delivered hours, not captured hours. The 14-hour weekly gap at baseline was invisible in the client’s dashboards.
Fix sync before buying anything. A 25 ms alignment tolerance is unreachable over ad-hoc Wi-Fi streaming without a shared clock.
Premium headsets are evaluation tools, not fleet tools. Quest 3 matched Vision Pro on demonstrations per operator-hour for standard manipulation families.
Match rig fidelity to task family. ALOHA-class cells for fine bimanual work, GELLO for arm tasks on owned robots, UMI for volume, one format across all of them.
Scale last. Every seat added before yield stabilizes multiplies your rejection rate.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

How long does it take to scale a robot data capture operation?

In this engagement, nine weeks from audit to a sustained 400 delivered hours per week. The first five weeks fixed synchronization, formats, and seat design; scaling seats was the fast part.

How much did the capture hardware cost at full scale?

About $118,000 in cumulative fleet capex for 42 seats: 12 Quest 3 teleop seats, 6 GELLO arm cells, 3 ALOHA-class stations, 20 UMI-style grippers, and a 6-camera RealSense D455 evaluation studio.

What QA rejection rate is normal for teleoperation data?

On a well-synchronized floor we hold 10 to 12 percent. Rates above 25 percent almost always trace to clock misalignment or ergonomic problems, not operator error.

Did the VLA team really not need Vision Pros?

Not for fleet capture. On pick, place, and insertion tasks, ~$500 Quest 3 seats matched the $3,499 Vision Pro on demonstrations per operator-hour. The Vision Pros stayed in service for gaze-labeled evaluation episodes, where their eye tracking earns the price.

Can we run this playbook in-house instead of using a vendor?

Yes; the phases (audit, re-spec, scale-out) and rules above are the playbook. The vendor case is throughput on demand without capex and hiring, which mattered here because of the five-month deadline.

Facing a similar deadline?

[Book a Demo] and we will run the throughput math on your task list, or start with the full guide to data capture hardware for embodied AI.

Comparing Data Capture Hardware for Embodied AI Approaches: Pros, Cons and Costs

Posted on July 28, 2026July 24, 2026 by Sainath Gupta

Our prediction, held at roughly 80 percent confidence: within two years, no serious VLA training stack will rely on a single capture approach, and the teams still arguing egocentric versus teleoperation will have lost the argument to “both.” That is an opinion, clearly flagged; the evidence behind it is not. Three camps currently give three confident, incompatible answers to the data collection question: strap cameras on humans, build teleop rigs, or ring the workspace with depth cameras. Each camp has papers proving its approach works, and none of the papers tell you what each approach costs per usable hour, or which one fits the model you are actually training.

The argument persists because these approaches produce different data, not the same data at different prices. That is the thesis this comparison argues: the four approaches are complements delivering different supervision signals, so the choice follows from your training objective. Egocentric human video has no robot actions in it. Teleoperation produces perfect action labels at real-time speed and real-time cost. Exocentric camera arrays see everything except the first-person view your humanoid will actually have. Choosing between them by hardware price alone is a category error.

This post compares the four capture approaches side by side: what hardware each needs, what data each yields, what each costs on our floors, and a decision matrix for matching approach to training objective. It condenses the hardware detail from our full guide to data capture hardware for embodied AI.

I run teleoperation systems at DexSet, where all four approaches operate in production, so the pros and cons below come from shift logs rather than spec sheets.

Key Takeaways – The four capture approaches are egocentric wearables, teleoperation, handheld demonstration devices (UMI-style), and multi-view exocentric arrays. They produce different modalities and are complements, not substitutes. – Egocentric capture is the cheapest per hour (a ~$400 GoPro seat) but carries no robot action labels; teleoperation is the most expensive but yields training-ready observation-action pairs. – Teleop hardware spans two orders of magnitude: ~$300 GELLO leader arms to $20,000+ ALOHA-class bimanual stations, with our delivered benchmarks at $28-60 per hour. – Most VLA training stacks want a blend: broad egocentric pretraining data plus targeted teleop fine-tuning data plus a small exo set for evaluation.

Approach 1: Egocentric Wearable Capture

Egocentric capture records first-person video and motion from devices worn by humans doing real tasks, no robot required. It is the approach behind Ego4D and Ego-Exo4D (arxiv.org/abs/2311.18259), and the standard source of large-scale pretraining data for humanoid perception.

Hardware options: GoPro head mounts (~$400, 5.3K/60, ~150 g), Meta Quest 3 (~$500, adds hand tracking and 6DoF pose), Apple Vision Pro ($3,499, adds eye tracking and LiDAR), and Project Aria research glasses (research access, best-in-class sensor sync at ~75 g).

Pros: lowest cost per hour of any approach; scales with people instead of robots; captures true task diversity in homes, kitchens, and warehouses; light devices sustain 6+ hour shifts.

Cons: no robot action labels, so it cannot train a policy by itself; consumer devices have weak cross-device sync; gaze and hand pose require the expensive end of the hardware range.

Approach 2: Teleoperation Capture

Teleoperation capture records a human operator controlling a real robot, producing synchronized observation-action pairs, which is exactly the supervision imitation learning consumes. The chain is direct: teleoperation produces demonstrations, demonstrations train ACT- and diffusion-policy-style models, and those methods feed VLA systems like OpenVLA and pi-0.

Hardware options: ALOHA-class leader-follower stations (~$20,000+ per bimanual cell, the platform behind ACT, arxiv.org/abs/2304.13705), GELLO printed leader arms (~$300 per arm on robots you already own, arxiv.org/abs/2309.13037), and VR teleop with a Quest 3 or Vision Pro streaming head and hand pose to the robot.

Pros: training-ready action labels at capture time; joint-space leader arms give fine bimanual control; output drops straight into LeRobot or HDF5 episode formats.

Cons: cost scales with robots, not just people; one real-time hour per data hour per rig; resets and calibration eat 30 to 40 percent of every shift; operator skill affects data quality.

Approach 3: Handheld Demonstration Devices

Handheld demonstration devices are robot-free grippers, UMI being the reference design, that a human carries through tasks while a wrist-mounted camera records; robot actions are recovered afterward via SLAM (arxiv.org/abs/2402.10329). The approach splits the difference between egocentric video and teleoperation.

Hardware: a 3D-printed UMI-style gripper plus a fisheye GoPro, roughly $400 per unit.

Pros: near-egocentric cost with usable action supervision; natural human motion, no retargeting latency; trivially parallel, ten grippers cost less than one robot arm.

Cons: no proprioception or force signal; action quality depends on SLAM post-processing, which becomes its own QA pipeline; single-gripper design limits bimanual tasks.

Approach 4: Multi-View Exocentric Arrays

Multi-view exocentric capture surrounds a workspace with synchronized third-person cameras, recording the scene from four to twelve angles at once. DROID standardized this pattern across labs (arxiv.org/abs/2403.12945), and it is the default source of evaluation and cross-view data.

Hardware options: Intel RealSense D435i (~$300-350) or D455 (~$420-450, wider baseline, hardware sync pin), Stereolabs ZED 2i (~$500, neural depth, needs an NVIDIA GPU per node), Luxonis OAK-D (~$250-350, on-device compute).

Pros: metric depth and full-scene coverage; hardware sync solves the alignment problem RealSense-style; fixed rigs need no operator wearables at all.

Cons: not the robot’s viewpoint; per-studio infrastructure (sync wiring, lighting, terabyte-per-shift storage) roughly doubles the camera bill; useless for capturing task diversity outside the cell.

What the Landmark Datasets Chose

The published datasets that shaped this field each committed to one of these approaches, and their choices are useful evidence. Ego4D collected thousands of hours of unscripted daily activity on head-mounted cameras because egocentric scale was the objective and action labels were not. ALOHA and Mobile ALOHA (arxiv.org/abs/2401.02117) went all-in on leader-follower teleoperation because fine bimanual action supervision was the entire point. DROID distributed identical RealSense-based exocentric rigs to dozens of labs to get multi-view consistency across sites. And Ego-Exo4D deliberately paired Aria glasses with stationary cameras, an explicit bet that ego plus exo beats either alone.

None of these teams picked their hardware by price. They picked by the supervision signal their research needed, then engineered the cost down. That is the right order of operations for a production pipeline too.

The Master Comparison Table

Approach	Typical Hardware	Capex per Seat/Cell	Data Yielded	Action Labels	Our Delivered Cost Benchmarks	Best For
Egocentric wearable	GoPro ~$400, Quest 3 ~$500, Vision Pro $3,499, Aria (research)	$400-4,000	First-person RGB, IMU, pose, gaze (device-dependent)	None	Lowest per hour (one third to one half of teleop)	Pretraining scale, humanoid viewpoint priors
Teleoperation	GELLO ~$300/arm, ALOHA ~$20k station, VR teleop $500-3,499 + robot	$600-30,000	Obs-action pairs, proprioception, 50 Hz control	Native, highest quality	$28-60/hr	Policy fine-tuning, bimanual manipulation
Handheld (UMI-style)	Printed gripper + GoPro, ~$400	$400-600	Wrist-cam RGB + SLAM-recovered trajectories	Recovered, no force/proprio	Between egocentric and teleop	Cheap single-arm demos at volume
Exocentric array	4-8x D455/ZED 2i/OAK-D	$2,500-12,000 per studio	Multi-view RGB-D, synced	None (pairs with teleop)	Studio amortizes across every episode captured inside it	Evaluation, cross-view consistency, world models

Decision Matrix: Matching Approach to Training Objective

The right capture approach is determined by what your model consumes, and the mapping is more stable than hardware fashion.

Pretraining a humanoid perception backbone: egocentric wearables first. Buy GoPro seats for volume; add Aria or Vision Pro only where gaze labels earn their cost.
Fine-tuning a VLA policy on manipulation tasks: teleoperation. GELLO on existing arms if you own robots; an ALOHA-class cell if bimanual dexterity is the product.
Thousands of single-arm demos on a budget: UMI-style handheld grippers, with a budgeted SLAM QA pass.
Evaluation sets and world-model data: an exo array of hardware-synced D455s wrapped around one teleop cell, so every episode is captured ego and exo simultaneously.
All of the above on a deadline: that blend is precisely what a data vendor amortizes across clients; the build-versus-buy arithmetic is in the full hardware guide.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

Which robot data collection approach is cheapest?

Egocentric wearable capture. A ~$400 GoPro head-mount seat produces first-person video at one third to one half the per-hour cost of teleoperation in our benchmarks, but it yields no robot action labels.

Is teleoperation data worth the higher cost?

For policy training, yes. Teleoperation is the only approach that natively records synchronized observation-action pairs with proprioception, which is the supervision imitation learning and VLA fine-tuning require.

What is a UMI gripper and when should I use one?

UMI is a handheld, robot-free gripper design with a wrist-mounted fisheye GoPro; actions are recovered by SLAM in post-processing. Use it when you need thousands of single-arm demonstrations at near-egocentric cost and can accept the absence of force and proprioception data.

Do I need an exocentric camera array if I already collect teleop data?

For evaluation and cross-view generalization, a small one helps. Wrapping four to six hardware-synced RealSense D455s around one teleop cell gives you paired ego-exo episodes, the structure Ego-Exo4D demonstrated matters for viewpoint generalization.

Can I mix data from all four approaches in one training set?

Yes, and most serious VLA stacks do: broad egocentric data for pretraining, teleop episodes for fine-tuning, and exo views for evaluation. The practical requirement is one shared episode format and honest per-source metadata.

Want to see the difference instead of reading about it?

[Download Sample Data] from all four capture approaches, or start with the complete guide to data capture hardware for embodied AI.

Why Data Capture Hardware for Embodied AI Is the Biggest Bottleneck in Physical AI

Posted on July 27, 2026July 24, 2026 by Enoch Pakanati

There is a rule on our capture floor that surprises every visitor: no rig purchase gets approved until someone writes down, and signs, the usable hours per shift it is expected to deliver. The rule exists because of a ratio most teams meet too late. A training cluster can process a million episodes a day; a capture floor produces a few hundred. That ratio, not model architecture, is the thesis of this post: capture hardware throughput is the binding constraint in physical AI, and almost every team we work with discovers it after they have already hired the ML engineers.

The gap exists for a structural reason. Language models scaled on data that already existed; the internet was captured for free, decades in advance, by billions of people. Robot data does not exist until someone straps on a camera or sits down at a teleoperation rig and creates it, one real-time hour at a time. Open X-Embodiment, the largest cross-robot dataset effort to date, pooled contributions from more than 20 institutions to assemble on the order of a million trajectories (arxiv.org/abs/2310.08864). GPT-class text corpora are trillions of tokens. The disparity is not a funding problem. It is a hardware throughput problem.

This post breaks down where the bottleneck actually sits: capture-rate physics, sync failure, operator throughput, and fleet reliability. You will get the numbers we use to size capture floors, and a checklist for finding your own pipeline’s binding constraint.

DexSet runs egocentric, teleoperation, and multi-view exocentric capture lines in production for robotics foundation model teams, so the failure modes below are ones we have paid for personally.

Key Takeaways – Robot data must be created in real time; one hour of demonstrations costs one hour of human effort plus overhead, which makes capture hardware throughput the binding constraint in physical AI. – The bottleneck has four layers: capture-rate physics, cross-sensor synchronization, operator ergonomics, and fleet reliability. Most teams only budget for the first. – In our production benchmarks, a well-run teleop cell yields 4 to 5 usable data hours per 8-hour operator shift; a badly specified one yields under 2. – Scaling is a parallelism problem: 10 cheap synchronized seats beat 2 premium ones on delivered hours per dollar, every time we have measured it.

The Bottleneck Is Physical, Not Computational

The core constraint in embodied AI is that demonstration data is generated at the speed of physical reality, one wall-clock hour per data hour, per rig. No compiler optimization changes this. A capture floor’s output is seats multiplied by shift hours multiplied by yield, and hardware choices control two of those three factors.

Consider the arithmetic that surprised one of our clients. They wanted 10,000 hours of teleop demonstrations in six months. At a realistic 4.5 usable hours per operator shift, that is roughly 2,200 shifts, or 18 rigs running double shifts for the full period with zero downtime. Their plan had budgeted 6 rigs. The model roadmap slipped a quarter before a single GPU was misused.

Where Hardware Throughput Actually Dies

Capture throughput is lost in four places, and each maps to a hardware decision made months earlier.

1. Capture-Rate Physics

Capture-rate physics is the raw sensor and storage arithmetic of a rig: resolution times frame rate times stream count. A six-camera RealSense D455 studio recording depth plus RGB produces on the order of a terabyte per shift before compression. Teams that bought cameras without buying the matching NVMe offload path spend the last hour of every shift copying files instead of capturing. The camera was $450; the missing storage architecture costs a data hour per day, forever.

2. Synchronization Failure

Synchronization failure is misalignment between sensor clocks, and it converts captured hours into rejected hours during QA. A 40 ms drift between headset pose and robot joint states is invisible to the operator and disqualifying for action-labeled training data. Hardware with a sync story (RealSense sync pins, Project Aria’s shared-clock VRS recordings) keeps yield around 90 percent in our pipelines. Hardware without one (consumer action cams in multi-view arrays) can silently push rejection rates past 30 percent, and you find out weeks later.

3. Operator Ergonomics

Operator ergonomics is the human sustainability of a rig across a full shift, and it is the most under-priced axis in hardware selection. A 600-plus-gram headset with an external battery is fine for a 20-minute demo and brutal at hour six. On our floors, moving long-shift egocentric work from headsets to ~150 g head-mounted action cameras raised sustained weekly output per operator by a third, purely because breaks got shorter and turnover dropped. ALOHA-style seated leader arms (arxiv.org/abs/2304.13705) similarly outlast handheld free-space teleop for repetitive tabletop tasks.

4. Fleet Reliability

Fleet reliability is the fraction of seats actually capturing at any moment, and it decays with device fragility and SDK churn. Consumer devices were not designed for 60-hour weeks: straps wear out, USB ports loosen, batteries lose capacity by month four, and a firmware update can strand a whole fleet’s capture app on a Tuesday morning. Budget spare units at 15 to 20 percent of fleet size and pin firmware versions the way you pin Python dependencies. A $3,499 spare hurts, which is one more argument for $500 seats.

There is a fifth, quieter drain worth naming: calibration and QA overhead. Every worn or moved device needs its extrinsics re-verified regularly, and every captured hour needs review before it counts. On a healthy floor this overhead runs 10 to 15 percent of shift time. On a floor that skips it, the overhead appears anyway, relocated into next month’s rejection rate.

The Throughput Table

The numbers below are typical ranges from our own capture operations. Treat them as planning figures.

Capture Setup	Capex per Seat	Usable Hours per 8h Shift	Common Bottleneck	Delivered Cost Range (our benchmarks)
GoPro head-mount egocentric (~$400)	$400-700	5.5-6.5	Offload bandwidth, battery swaps	Lowest per hour of any modality
Quest 3 VR teleop (~$500 + robot)	$1,200-5,500	4-5	Passthrough latency fatigue, resets	$28-45/hr
Vision Pro VR teleop ($3,499 + robot)	$4,500-8,000	3.5-4.5	Comfort, battery, spare cost	$40-60/hr
ALOHA-class bimanual (~$20k station)	$22,000-30,000	4-5	Task resets, arm maintenance	$35-60/hr
GELLO leaders on existing arms (~$300/arm)	$600-1,000 incremental	4-5	Follower robot availability	$28-45/hr
UMI handheld grippers (~$400)	$400-600	5-6	SLAM post-processing QA	Below teleop, above raw egocentric

Read the table columns together, not in isolation. The cheapest capex row is not automatically the cheapest delivered hour once yield and QA rejection enter, but the pattern holds: parallel cheap seats with a real sync plan dominate small fleets of premium hardware.

How to Scale Past the Bottleneck

Scaling data capture hardware for embodied AI means adding synchronized parallel seats while holding per-seat yield constant, and the order of operations matters.

Instrument yield first. Measure usable hours per shift per seat before buying anything. Most teams cannot answer this today.
Fix sync before adding seats. Ten unsynced rigs scale your rejection rate, not your dataset.
Standardize on one episode format. LeRobot’s dataset format (github.com/huggingface/lerobot) or a fixed HDF5 schema; converters eat engineer-weeks.
Buy for the shift, not the demo. Weigh the device, check the battery swap time, and price spares at 15-20 percent of fleet.
Then parallelize. Cheap identical seats, one calibration procedure, one QA gate.

The order matters more than the shopping list. Teams that parallelize first and fix sync second end up owning ten rigs that produce ten streams of mutually misaligned data, and the “scaling” project quietly becomes a salvage project.

Or skip the capital expenditure entirely: this bottleneck is exactly why data vendors exist. A vendor amortizes rigs, operators, calibration routines, and QA gates across many clients, which is how delivered hours can cost less than an in-house floor’s fully loaded rate even before you count the hiring time. The build-versus-buy math is covered in our full guide to data capture hardware for embodied AI, including per-device comparison tables and a decision matrix.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

Why is data capture the bottleneck in physical AI rather than compute?

Because robot demonstrations must be created in real time by humans and hardware, one wall-clock hour per data hour per rig. Compute scales by purchase order; capture scales only by adding synchronized seats and operators.

How many usable data hours does one teleoperation rig produce per day?

In our production benchmarks, a well-specified teleop cell yields 4 to 5 usable hours per 8-hour shift after resets, calibration, and QA rejection. Poorly synchronized or ergonomically bad rigs can drop below 2.

What is the fastest way to scale robot data collection?

Fix synchronization and episode formats first, then add cheap parallel seats. Ten ~$500 synchronized capture seats reliably outdeliver two premium seats on hours per dollar.

How much data do VLA models need?

Published systems give the scale: Open X-Embodiment pooled roughly one million trajectories across institutions, and fine-tuning a VLA for a task family typically consumes hundreds to thousands of targeted demonstrations. The practical answer is more than one team’s ad-hoc rig produces, which is why capture throughput planning matters.

Does buying more expensive capture hardware increase throughput?

Not by itself. Throughput follows seats, sync, ergonomics, and reliability. A $3,499 headset that fatigues operators faster than a $500 one lowers delivered hours per dollar despite better optics.

Sizing a capture floor?

Start with the full guide to data capture hardware for embodied AI, or book a demo and we will walk you through the throughput math on your task list with real DexSet sample data.

The Complete Guide to Data Capture Hardware for Embodied AI (2026)

Posted on July 24, 2026July 24, 2026 by Sainath Gupta

A Head of Data on a scoping call last quarter asked us which single capture device her team should standardize on, and for the first time in months, we had no quick answer. We had run every device she named in production. The problem was the question: it has no device-shaped answer, because a capture device is not a purchase, it is a commitment. The hardware you choose locks your pipeline into what that device can timestamp, sync, and survive, and teams that fail on data usually failed at that moment, months before anyone touched a model. That is this guide’s thesis: capture hardware is a data-strategy decision, judged on seven axes and on cost per delivered hour, never on spec sheets.

The reason the question stumps buyers is that capture hardware for embodied AI sits in a gap between consumer electronics and lab equipment. A Quest 3 was built for gaming, not for exporting synchronized RGB, IMU, and hand pose at training quality. A RealSense was built for prototyping, not for 40 units running ten hours a day. Vendor spec sheets answer none of the questions that matter for foundation model training, so teams learn the hard lessons after the purchase order clears.

This guide gives you the full picture: every major hardware category (headsets, research glasses, action cameras, stereo depth cameras, teleoperation rigs, handheld grippers, tactile sensors), compared on the seven axes that decide whether your dataset trains a VLA model or fills a hard drive. You will leave with real prices, a cost-per-hour model, and a decision matrix mapped to the three capture strategies that matter: egocentric human data, teleoperation, and multi-view exocentric capture.

At DexSet we run these devices in production. Our teams operate teleop cells, headset fleets, and multi-camera capture studios that deliver egocentric, exocentric, mono, and stereo data to robotics foundation model teams. Every number in this guide comes from our own rigs, published research, or a linked primary source.

TL;DR: Key Takeaways – Data capture hardware for embodied AI spans four categories: wearable egocentric devices ($350 to $3,499), stereo depth cameras ($250 to $500 per unit), teleoperation rigs ($300 per arm to $20,000+ per station), and tactile sensors. – The seven evaluation axes are price, resolution/fps, depth quality, sync capability, wearability, SDK maturity, and data formats. Sync capability is the axis teams most often ignore and most often regret. – A Meta Quest 3 at ~$500 covers most teleoperation UI needs; the Apple Vision Pro at $3,499 earns its price only for eye tracking and high-resolution passthrough research. – Teleop capex is lower than most CTOs assume: a GELLO leader arm costs roughly $300 per arm, against $20,000+ for a full ALOHA-class bimanual station. – In our production benchmarks, delivered teleop data runs $28 to $60 per hour depending on rig and task complexity; egocentric human data runs meaningfully cheaper per hour but trains different capabilities.

What Is Data Capture Hardware for Embodied AI?

Data capture hardware for embodied AI is the set of physical devices, wearables, cameras, teleoperation interfaces, and sensors, used to record the visual, spatial, inertial, tactile, and action data that trains robots and vision-language-action (VLA) models. It is the acquisition layer of the robot learning stack: everything between the physical world and your training dataset.

The category matters because embodied AI models learn from demonstrations, not from text scraped off the internet. A VLA model like OpenVLA (arxiv.org/abs/2406.09246) or Physical Intelligence’s pi-0 (arxiv.org/abs/2410.24164) consumes synchronized streams of camera frames, proprioception, and actions. The hardware that records those streams sets a hard ceiling on model quality. No amount of post-processing recovers depth a sensor never measured or a timestamp a device never wrote.

Core Concepts and Data Modalities

Data modalities are the distinct sensor streams a capture device records, and every hardware decision starts by listing which modalities your model actually needs. The main ones for embodied AI are RGB video, stereo pairs, depth, IMU, eye gaze, hand and body pose, robot proprioception, and tactile signal.

Mono vs. Stereo RGB

Mono RGB is a single camera stream; stereo RGB is a calibrated pair that allows depth to be recovered by triangulation. Mono is cheaper and lighter and dominates large egocentric datasets like Ego4D (arxiv.org/abs/2110.07058). Stereo gives your model, or your post-processing pipeline, metric 3D structure, which matters for manipulation policies that must reason about distance to a grasp point. On our rigs, stereo roughly doubles storage and calibration workload, so we specify it only when the downstream model consumes it.

Depth

Depth is a per-pixel distance measurement, produced either by active sensors (structured light, time-of-flight) or by stereo matching. Active depth from a RealSense-class sensor works well indoors at 0.3 to 6 meters but degrades in sunlight and on reflective surfaces. Neural stereo depth (ZED 2i class) extends range but requires a GPU at capture time or in post.

IMU and Time Synchronization

An IMU (inertial measurement unit) records acceleration and angular velocity, typically at 200 Hz to 1,000 Hz, and it is only useful if its clock aligns with the camera clock. Cross-device sync is the silent killer of embodied AI datasets: a 40 ms offset between video and proprioception is invisible in playback and fatal for action-labeled training data. Devices differ enormously here. Project Aria timestamps all sensors on a shared clock; a GoPro gives you a self-consistent file but no native way to sync with a robot’s ROS 2 clock without an external signal.

Proprioception and Action Data

Proprioception is the robot’s own joint state (positions, velocities, torques), and action data is the command stream sent to the robot. These come from the robot or the teleoperation interface, not from cameras, which is why teleop rigs are capture hardware in their own right. Imitation learning methods like ACT, introduced with the ALOHA rig (arxiv.org/abs/2304.13705), train directly on this stream. The entity chain to keep straight: teleoperation produces demonstrations, demonstrations feed imitation learning, imitation learning trains the VLA policies that run on robots.

Tactile

Tactile sensing records contact geometry and force at the gripper, usually via vision-based sensors in the GelSight style, where a camera watches a deformable gel pad. Tactile is the newest modality in production pipelines and the least standardized; we cover the hardware below.

The Four Hardware Categories

Capture hardware for embodied AI falls into four categories: wearable egocentric devices, stereo and depth cameras for exocentric capture, teleoperation rigs, and handheld or tactile interfaces. Most production pipelines combine at least two.

Category 1: Wearable Egocentric Devices

Wearable egocentric devices are head-mounted cameras and headsets that record the world from a human’s first-person viewpoint, the same viewpoint a humanoid robot’s head camera will have. This category powers egocentric pretraining datasets like Ego-Exo4D (arxiv.org/abs/2311.18259) and doubles as the operator interface for immersive teleoperation systems like Open-TeleVision (arxiv.org/abs/2407.01512).

Device	Price	Sensors & Resolution	Depth	Sync	Wearability (8h shift)	SDK Maturity	Native Data Formats
Meta Quest 3	~$500	Color passthrough cameras, 6DoF inside-out tracking, hand tracking, IMU	Passthrough depth estimate, not export-grade	Good within headset; WebXR/OpenXR timestamps for teleop	Moderate; 515 g, front-heavy after ~2h	Strong (OpenXR, Meta SDK, large dev community)	App-level capture; pose + hand joints via OpenXR
Apple Vision Pro	$3,499	High-res micro-OLED displays, high-quality color passthrough, eye tracking, hand tracking, LiDAR	LiDAR scene depth	Excellent within device; ARKit timestamps	Poor for long shifts; 600-650 g, external battery	Strong but restrictive (visionOS, ARKit); raw camera access gated	ARKit poses, hand skeletons; limited raw sensor export
Project Aria / Aria Gen 2	Research access (not retail)	RGB camera, 2 mono SLAM cameras, eye tracking cameras, spatial mics, dual IMUs (Gen 2 adds PPG, improved compute)	No active depth; SLAM-derived	Best in class; all sensors on shared clock, VRS format	Excellent; ~75 g glasses form factor	Good (Project Aria Tools, MPS cloud services)	VRS files, MPS trajectories, gaze (projectaria.com)
GoPro head mount (Hero-class)	~$400	Single wide FOV RGB, 5.3K/60 or 4K/120, IMU telemetry	None	Weak across devices; GPMF telemetry internal only	Good; ~150 g with mount	Moderate (GPMF parsers, no live SDK)	MP4 + GPMF metadata track

The pattern to notice: the $3,499 device is not seven times better than the $500 device for data capture. Vision Pro wins when you need eye gaze and the highest-fidelity passthrough for perception research or premium teleop. Quest 3 wins on fleet economics: at our scale, one damaged Vision Pro costs the same as seven Quest 3 replacements. Aria wins on data quality per gram but is research-access only, which makes it a partnership decision rather than a purchasing decision. The GoPro remains the workhorse for raw egocentric video volume, and it is the camera inside the UMI gripper stack discussed below.

Category 2: Stereo Depth Cameras for Exocentric Capture

Stereo depth cameras are fixed or tripod-mounted sensors that record a scene from outside the actor, providing the third-person views used in multi-view exocentric datasets and in robot workspace monitoring. Datasets like DROID (arxiv.org/abs/2403.12945) standardized on this class of sensor.

Device	Price	RGB / Depth Resolution & FPS	Depth Quality	Sync Capability	SDK Maturity	Data Formats
Intel RealSense D435i	~$300-350	Depth 1280x720 @ up to 90 fps; RGB 1080p @ 30; IMU	Good 0.3-3 m; global-shutter depth sensors	Hardware sync pin across units	Mature (librealsense, ROS 2 wrapper)	ROS 2 bags, .bag/.ply, raw streams
Intel RealSense D455	~$420-450	Depth 1280x720 @ 90 fps; wider 95 mm baseline; global-shutter RGB; IMU	Better than D435i at 4-6 m range	Hardware sync pin	Mature (same stack)	Same as D435i
Stereolabs ZED 2i	~$500	Stereo 2K @ 15, 1080p @ 30, 720p @ 60; IMU, barometer, magnetometer	Neural stereo depth, usable 0.3-20 m	Software sync; multi-cam via ZED Hub	Mature but GPU-dependent (ZED SDK, needs NVIDIA)	SVO recordings, ROS 2 wrapper
Luxonis OAK-D	~$250-350	4K RGB + 2x mono stereo pair; on-device compute	Good short-range stereo depth	Hardware FSYNC on some variants	Growing (DepthAI, ROS 2 driver)	DepthAI streams, ROS 2

For multi-view exo capture, the deciding axis is sync. RealSense’s hardware sync pin lets you drive eight cameras off one trigger, which is why it still dominates academic capture cells despite Intel’s on-again-off-again commitment to the product line. ZED 2i buys you longer range and better outdoor behavior at the cost of an NVIDIA GPU per capture node. OAK-D is the value play when you want on-device neural inference to filter or annotate at capture time.

Category 3: Teleoperation Rigs

Teleoperation rigs are the leader-follower arm pairs, exoskeletons, and VR-based control systems through which a human operator produces robot action data. This is the category where hardware choice most directly sets your cost per demonstration.

System	Price	What It Is	Action Fidelity	Operator Fatigue	SDK / Ecosystem	Data Formats
ALOHA / ALOHA 2 rig	~$20,000+ per bimanual station	Two leader + two follower arms, joint-space mapping, 50 Hz	High; fine bimanual tasks	Moderate; seated, arms supported	Strong research ecosystem (ACT, Mobile ALOHA, LeRobot support)	HDF5 episodes; LeRobot dataset format
GELLO	~$300 per arm (plus follower robot)	3D-printed, Dynamixel-based kinematic-twin leader arm (arxiv.org/abs/2309.13037)	High for the matched follower	Low-moderate	Open source, LeRobot-compatible	Joint streams to ROS 2 / LeRobot
UMI gripper	~$400 build (printed gripper + GoPro)	Handheld gripper with wrist-mounted fisheye GoPro; robot-free demonstrations (arxiv.org/abs/2402.10329)	Good; SLAM-recovered actions, no proprioception	Low; natural human motion	Open source, active community	MP4 + GPMF, post-processed to episodes
VR teleop (Quest 3 / Vision Pro + arms)	$500-$3,499 + robot	Headset streams head/hand pose to robot controller (Open-TeleVision style)	Good; depends on retargeting	Moderate; passthrough latency drives fatigue	Growing; OpenXR + custom stacks	Pose streams + robot logs
Tactile (DIGIT / GelSight-style)	A few hundred dollars per fingertip (DIGIT-class); research-grade GelSight units cost thousands	Vision-based tactile sensor in gripper fingertips	Adds contact-rich signal	N/A (robot-mounted)	Moderate (open DIGIT designs; PyTouch-era tooling)	Compressed tactile image streams

The $20,000 ALOHA figure and the $300 GELLO figure describe different points on the same curve, not competing products. ALOHA is a complete bimanual station including four arms. GELLO is a leader-arm design you bolt onto follower robots you already own. UMI removes the robot from data collection entirely, which is why it produces the cheapest demonstrations and the weakest proprioception. Every serious pipeline we run mixes at least two of these.

Category 4: Multi-View Exocentric Studios

A multi-view exocentric studio is an array of four to twelve synchronized cameras surrounding a workspace, recording the same manipulation from multiple third-person angles. Ego-Exo4D paired Aria glasses with four to five stationary GoPros per scene precisely because models trained on both viewpoints generalize better than either alone. Budget $2,500 to $6,000 per studio in cameras and mounts, then double it for the sync, lighting, and storage infrastructure that spec sheets never mention.

Decision Matrix: Which Hardware for Which Use Case

The right hardware follows from the capture strategy, and there are three dominant strategies: egocentric human data, teleoperation, and multi-view exocentric capture.

Use Case	First Choice	Budget Alternative	Premium Option	Avoid
Egocentric human data at scale (pretraining)	GoPro head mounts (~$400/operator)	Used Hero-class cameras	Project Aria via research partnership	Vision Pro (cost, battery, comfort)
Egocentric with gaze + trajectories	Project Aria / Aria Gen 2	Quest 3 with logging app	Aria Gen 2	GoPro (no gaze, no shared clock)
Immersive teleoperation UI	Quest 3 (~$500)	Quest 3	Vision Pro (eye tracking, passthrough fidelity)	GoPro (not a display)
Bimanual fine manipulation demos	ALOHA-class station (~$20k)	GELLO leaders on existing arms (~$300/arm)	ALOHA 2 with tactile fingertips	UMI (no bimanual joint coupling)
Cheap single-arm demonstrations at volume	UMI grippers (~$400 each)	GELLO + one follower arm	ALOHA station	Vision Pro teleop (capex per seat)
Multi-view exo studio	4-8x RealSense D455, hardware-synced	OAK-D array	ZED 2i array with GPU nodes	Unsynced consumer webcams
Contact-rich tasks	DIGIT-class fingertips on any rig above	Single tactile finger + RGB	GelSight-style research sensors	Skipping tactile, then labeling contact by eye

Cost, Economics, and ROI

Capture hardware economics are dominated by throughput and operator time, not by device price, so the correct unit of analysis is cost per delivered data hour. This is the section competitors leave out, so here are the numbers we actually use.

Capex per seat, typical ranges:

Egocentric human capture seat: $400 to $700 (camera, mount, spare batteries, storage cards)
VR teleop seat: $1,200 to $5,500 (headset, workstation, networking; excludes the robot)
GELLO teleop seat: $600 to $1,000 on top of an existing follower arm
ALOHA-class bimanual cell: $22,000 to $30,000 fully commissioned
Exo studio (6 cameras): $2,500 to $6,000 in sensors, roughly the same again in sync, mounting, lighting, and storage

Cost per delivered hour, our production benchmarks: teleoperation data lands at $28 to $60 per hour depending on rig type, task complexity, and QA depth. Egocentric human data typically lands at one third to one half of that, because a human wearing a camera needs no robot, no retargeting stack, and far less resets-per-hour overhead. These are the typical ranges we see across our own pipelines; treat them as planning numbers, not quotes.

The ROI logic is straightforward. A $20,000 ALOHA cell running two shifts amortizes to under $2 per data hour within a year; the operators cost ten times that. Buying cheaper hardware that cuts operator throughput by 20 percent is a false economy. Buying $3,499 headsets when $500 ones sustain the same demonstrations per hour is the opposite mistake.

Case Study: A Humanoid Foundation Model Team’s Capture Stack

The proof is in a deployment, so here is an anonymized configuration from our own work. A humanoid foundation model team came to us needing three data types simultaneously: egocentric human video for pretraining, bimanual teleop demonstrations for fine-tuning, and multi-view exo recordings for evaluation.

We stood up three lines. Line one: 30 GoPro head-mount seats for kitchen and warehouse egocentric capture, delivering several hundred hours per week. Line two: four ALOHA-class stations plus six GELLO-equipped single-arm cells, producing task-labeled demonstrations in LeRobot format (github.com/huggingface/lerobot). Line three: a six-camera RealSense D455 studio, hardware-synced, wrapped around one teleop cell for paired ego-exo evaluation episodes.

Two hardware lessons generalized. First, the D455’s wider baseline halved our depth QA rejections at 2 to 4 meters compared with the D435i units we started with. Second, moving VR teleop from Vision Pro pilots to Quest 3 fleet units cut capex per seat by 85 percent with no measurable drop in demonstrations per operator-hour on pick-and-place classes of tasks. The Vision Pros stayed in service for the subset of tasks where gaze labels earned their cost.

Get the RFP Scorecard

A hardware decision this size deserves a structured evaluation, so we built one you can reuse. The DexSet Data Capture Hardware RFP Scorecard is a one-page matrix that scores any device or vendor on the seven axes in this guide (price, resolution/fps, depth quality, sync, wearability, SDK maturity, data formats) plus delivery terms and QA guarantees. Teams use it to compare quotes on equal footing instead of vendor spec sheets. Download it below, and if you want the comparison done for you, that is literally our job.

Related reading from this series:

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

What is data capture hardware for embodied AI?

Data capture hardware for embodied AI is the set of wearable cameras, headsets, depth sensors, teleoperation rigs, and tactile sensors used to record the synchronized visual, inertial, action, and contact data that trains robot policies and vision-language-action models.

Is the Apple Vision Pro worth it over the Meta Quest 3 for robotics data capture?

Only for specific needs. The Vision Pro’s eye tracking and passthrough fidelity justify its $3,499 price for gaze-labeled data and premium teleoperation research. For fleet teleop and general egocentric capture, the ~$500 Quest 3 delivers comparable demonstrations per operator-hour at one seventh the capex.

How much does a teleoperation rig cost?

Between $300 and $30,000 depending on class. A GELLO leader arm costs roughly $300 per arm on top of a follower robot you already own; a UMI handheld gripper builds for about $400; a complete ALOHA-class bimanual station runs $20,000 or more fully commissioned.

What is the difference between egocentric and exocentric data capture?

Egocentric capture records from the actor’s first-person viewpoint using head-mounted devices; exocentric capture records the actor from external, third-person cameras. Datasets like Ego-Exo4D pair both because policies trained on combined viewpoints generalize better.

Why does time synchronization matter so much in capture hardware?

Because embodied AI models train on aligned observation-action pairs. A clock offset of tens of milliseconds between camera, IMU, and robot proprioception silently corrupts action labels, and it cannot be fixed after the fact without shared timestamps or an external sync signal.

Which stereo depth camera is best for a multi-camera robotics capture studio?

For hardware-synchronized multi-view rigs, the Intel RealSense D455 (~$420-450) is the default choice because of its sync pin, 95 mm baseline, and mature ROS 2 support. The ZED 2i suits longer-range or outdoor scenes if each node has an NVIDIA GPU.

Ready to skip the hardware learning curve?

DexSet already runs the rigs in this guide at production scale. [Download the RFP Scorecard] to evaluate vendors on equal terms, or [Book a Demo] to see sample egocentric, teleop, and multi-view datasets from these exact devices. [Download Sample Data] is one click away.

5 Hidden Challenges in Teleoperation Data Collection (and How to Solve Them)

Posted on July 23, 2026July 21, 2026 by Sainath Gupta

One house rule on our collection floor reliably surprises visitors: no episode ships the day it is collected, however clean it looks on the monitor, because every episode waits for the overnight QA gate first. The rule exists because the visible challenges in teleoperation, buying rigs, hiring operators, picking a data format, are the ones teams plan for. The hidden challenges surface at episode 5,000, when the dataset is half-collected, the deadline is fixed, and the training run just exposed a flaw that has been silently poisoning every episode since week one.

That is the thesis of this post: the failures that actually kill teleoperation datasets are invisible in any single episode and only show up in aggregate, so the fix is instrumentation at ingest, not heroics at training time. Each problem looks like a minor engineering detail until it compounds. A 40-millisecond timestamp offset is invisible in a demo video. Spread across 10,000 episodes, it is the difference between a 70 percent policy and a 45 percent one.

Below are the five failures we see most often across the programs we run and audit, each with the fix we actually use. If you are new to the space, start with the complete teleoperation data collection guide and come back.

Key Takeaways – Timestamp skew between 50 Hz actions and 30 fps cameras is the top QA rejection cause we see. Gate at 33 ms, at ingest. – Operator churn is a 3x throughput event, not an HR footnote: trained operators produce 25 to 40 usable episodes per hour versus 8 to 12 for novices. – Reset drift quietly narrows your state distribution; randomization zones fix it. – QA discovered at training time is loss; QA at ingest is feedback. Expect 10 to 30 percent rejection either way, so choose when you find out. – Format decisions are nearly irreversible after 10,000 episodes. Adopt LeRobot conventions on day one.

1. Timestamp Skew: The 50 Hz vs 30 fps Trap

Timestamp skew is the misalignment between a robot’s high-rate action stream and its lower-rate camera streams, and it is the single most common reason episodes fail our QA gates. Actions log at 50 Hz; cameras deliver 30 fps; USB buffering, clock drift between machines, and software timestamps assigned at receipt rather than exposure can push streams 30 to 100 ms apart. A policy trained on skewed data learns that consequences precede actions. It will move confidently and miss.

The fix. Hardware timestamps at the sensor wherever the camera supports it; a single shared monotonic clock otherwise; and an ingest gate that rejects any episode with estimated inter-stream skew above one camera frame (33 ms). We also log a clap-style sync event at the start of each shift as a cheap ground-truth check. Action chunking in ACT-style training tolerates per-step jitter, but no algorithm recovers from systematic offset (arxiv.org/abs/2304.13705).

2. Operator Churn: The Throughput Cliff

Operator churn is the loss of trained teleoperators before their throughput has paid back their ramp cost. The ramp curve is steep: 8 to 12 usable episodes per hour in the first two weeks, 25 to 40 after certification. Every departure resets a station to novice yield for two to three weeks, which means a floor with 20 percent monthly churn is quietly running at two-thirds of its rated capacity.

The fix. Treat operators as skilled technicians, because they are. Certification gates with pay progression, per-operator quality dashboards (people improve what they can see), 45-minute session blocks on VR rigs to manage fatigue, and rig ergonomics worth caring about. Leader-follower stations retain operators better than SpaceMouse benches for the simple reason that the work feels like skill rather than struggle.

3. Reset Drift: The Shrinking State Distribution

Reset drift is the gradual narrowing of initial-state diversity as operators unconsciously converge on the fastest reset, placing objects in the same pose, episode after episode. Nobody notices because every individual episode passes QA. The policy trains beautifully, evaluates beautifully on the collection table, then fails the moment an object appears five centimeters left of its usual spot. This is how teams end up with 10,000 episodes of one initial state.

The fix. Marked randomization zones on the workspace, task cards that specify pose ranges, and a distribution monitor in the QA pipeline that plots initial object positions per task per week. When the scatter tightens, the floor lead hears about it the next morning. Datasets like DROID earned their generalization value from scene diversity, 564 scenes across 76,000 episodes, not from episode count alone (arxiv.org/abs/2403.12945).

4. Silent QA Loss: Finding Out at Training Time

Silent QA loss is the gap between episodes collected and episodes actually usable, discovered only when someone trains on the data weeks later. Rejection is not the problem; 10 to 30 percent rejection is normal and we publish those numbers. The problem is timing. Rejection found at ingest becomes operator feedback and a corrected process by tomorrow. Rejection found at training time is a hole in the dataset, a slipped deadline, and no way to know which of the last six weeks went wrong.

The fix. Nightly automated gates on every episode: sync skew, dropped frames, joint-trajectory jerk thresholds, gripper event sanity, and success-label verification against a sampled human review. Route failures to the specific operator and rig. Our floors run this as a standing morning report, and it is the highest-ROI engineering month a data program can spend.

5. Format Lock-In: The Decision You Make Once

Format lock-in is the practical irreversibility of episode schema decisions once collection is underway, because re-processing tens of thousands of episodes costs more than most teams will ever approve. Missing per-episode metadata, camera intrinsics never recorded, action spaces defined ambiguously (commanded versus measured joint positions, unlabeled): each omission is trivial to fix on day one and a migration project at episode 20,000.

The fix. Adopt community conventions before the first production episode: LeRobot dataset format for training and sharing (github.com/huggingface/lerobot), HDF5 or ROS 2 bags at capture (docs.ros.org), both commanded and measured joint states recorded, camera intrinsics and extrinsics stored per rig per calibration, and schema versioning from v0. Fifteen minutes of schema review now, or fifteen days of migration later.

How to Audit Your Own Pipeline This Week

A pipeline audit is a half-day exercise that estimates your exposure to all five hidden challenges using data you already have. No new tooling required, in most cases.

Pull ten random episodes from last month and compute the maximum camera-to-control timestamp offset per episode; anything over 33 ms means challenge one is live in your dataset right now. Ask your floor lead for usable-episodes-per-hour by operator tenure; if you cannot get the number, that absence is itself the finding. Scatter-plot initial object positions for your highest-volume task across weeks one and four of collection; visibly tightening scatter is reset drift in progress. Check when rejection is discovered: if the answer involves a training run, you have silent QA loss by definition. And finally, hand an episode file to an engineer who has never seen your schema and time how long it takes them to load actions, states, and calibrated camera streams; over thirty minutes predicts format pain at scale.

Every one of these checks has caught a live problem on programs we have audited, including our own. The five challenges are not exotic failure modes; they are the default state of a pipeline that nobody has audited yet.

The Checklist

Hidden challenge	Detection signal	Fix	Cost of ignoring
Timestamp skew	Skew estimate > 33 ms at ingest	Hardware timestamps + sync gate	Policies that act early/late; top rejection cause
Operator churn	Throughput below 20 eps/hr floor-wide	Certification, pay progression, ergonomics	Floor runs at ~2/3 capacity
Reset drift	Initial-state scatter tightening week over week	Randomization zones + distribution monitor	Policy memorizes one layout
Silent QA loss	Rejection discovered at training time	Nightly ingest gates + operator feedback	10 to 30% of budget wasted invisibly
Format lock-in	Schema questions at training time	LeRobot conventions + schema v0	Migration project at episode 20k

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

What is the most common QA failure in teleoperation data?

Inter-stream timestamp skew, typically between 50 Hz action logs and 30 fps camera streams. We gate at 33 ms of estimated skew and it remains the top rejection reason on new rigs.

How much teleoperation data fails quality checks?

Expect 20 to 30 percent rejection during the first two weeks on a new task or rig, settling to 10 to 15 percent in a mature pipeline. Budget the buffer into episode targets.

Why do teleoperation-trained policies fail on new object positions?

Usually reset drift: operators converge on repeated initial states during collection, so the dataset covers a narrow state distribution. Randomization zones and initial-state monitoring prevent it.

When should QA run on teleoperation episodes?

At ingest, nightly, never later. Rejection found at ingest becomes next-day operator feedback; rejection found at training time is unrecoverable loss weeks after the fact.

What data format should a new teleoperation program use?

LeRobot dataset conventions for training and sharing, with HDF5 or ROS 2 bags at capture. Record both commanded and measured joint states, camera calibration per rig, and version the schema from the start.

Case Study: Scaling Teleoperation Data for a VLA Model, 1,900 to 12,600 Episodes in Ten Weeks

Posted on July 22, 2026July 21, 2026 by Enoch Pakanati

A head of data once asked us which of their 1,900 collected episodes had actually made their policy better, and we could not tell them; neither could they, because nothing in their pipeline had ever connected episodes to outcomes. That question came from the team this case study is about: eight weeks into their own collection effort, 1,900 episodes against a 12,000-episode target, a 34 percent QA rejection rate, and a training deadline that had already slipped once. Their ML team was strong. Their collection operation was two engineers moonlighting as floor managers, and it showed.

This is the common failure shape, and it is the thesis of this case study. Collection stalls not because anyone lacks skill but because demonstration data is an operations problem wearing an ML costume: staffing, ergonomics, and QA gating, none of which appear in the papers. The papers do tell you what is possible; ACT hit 80 to 90 percent success from about 50 clean demos per task (arxiv.org/abs/2304.13705). Getting 12,000 clean demos across 40 tasks is a different discipline.

Here is exactly what we changed, week by week, including the two things that failed. Use it as a template for your own program, or as an argument for not running one in-house.

Key Takeaways – Baseline: 1,900 episodes in 8 weeks, 34 percent rejection. Outcome: 12,600 QA-passed episodes in 10 weeks, 12 percent final rejection. – Six leader-follower stations plus a two-week operator certification program did most of the work. – Automated ingest QA (sync skew, smoothness, completion gates) cut silent data loss to near zero. – Client’s ACT baseline improved from 41 to 68 percent average success across their internal 40-task eval. – Peak steady-state: about 1,400 usable episodes per week from 8 operators.

The Starting Point: Why 1,900 Episodes Took Eight Weeks

The client’s bottleneck was a stack of small losses, not one big one. Their audit told the story: two VR stations shared among rotating volunteers, no defined operator roles, no reset protocol, and QA performed manually by an ML engineer, at training time, weeks after collection. Rejection ran 34 percent, dominated by timestamp skew between their 50 Hz action logs and 30 fps cameras, plus inconsistent initial states that made half the “passing” episodes weak training signal anyway.

Their effective yield was under 60 usable episodes per day. At that rate the 12,000-episode target sat more than eight months out.

What We Changed

The intervention had four parts: stations, people, protocol, and gates. In order of impact:

1. Stations. We replaced the two shared VR rigs with six bimanual leader-follower stations (GELLO-class leaders on the client’s arms, roughly $300 per leader plus fabrication), keeping one VR station for the mobile-base tasks where it fit better. Joint-level control cut trajectory hesitancy immediately and shortened operator ramp.

2. People. Eight dedicated operators on a two-week certification program: task cards, reset drills, and a throughput gate (20 usable episodes per hour on a reference task) before touching production tasks. Our standard ramp curve applied; novices started at 8 to 12 usable episodes per hour and certified operators stabilized at 25 to 40.

3. Protocol. Written task cards for all 40 tasks, marked randomization zones for object placement, success criteria defined at collection time, and per-episode metadata (task, operator, rig, success) attached at the source.

4. Gates. Automated ingest QA running nightly: inter-stream sync skew over 33 ms flagged, dropped-frame counts, jerk thresholds on joint trajectories, and completion labels verified against a sampled human review. Failures routed back to the floor as next-morning feedback per operator, which mattered as much as the gate itself.

The Week-by-Week Numbers

Weekly usable output tells the ramp story better than any narrative.

Week	Usable episodes	Rejection rate	Notes
1	310	29%	Station bring-up; operators in certification
2	540	26%	First cohort certified; jig redesign after reset failures
3	980	19%	All 8 operators on production tasks
4	1,390	15%	Steady state reached
5 to 9	1,300 to 1,450/wk	11 to 14%	Two task cards rewritten after eval feedback
10	1,280	12%	Target passed: 12,600 total QA-passed

Two failures worth admitting. In week two, our first reset jig for a deformable-object task increased reset time instead of cutting it; operators beat the jig by hand and we scrapped it. And one of the 40 tasks, bimanual jar opening, ran a stubborn 25 percent rejection until week six, when slowing the commanded demonstration pace fixed what better hardware had not.

Did the Data Actually Help? Policy Results

The point of episode counts is policy performance, so the client re-trained their ACT baseline on the new corpus at week eleven. Average success across their internal 40-task evaluation went from 41 percent (trained on their original 1,900 mixed-quality episodes) to 68 percent, with the largest gains on precision tasks where the old VR data had been noisiest. Same architecture, same hyperparameter budget, different data. Their team’s summary, which we will keep quoting: “We spent six months on model changes for less improvement than ten weeks of better data.”

That is one program, not a controlled study, and we flag it as such. But it matches the pattern across our floors and the published record: demonstration quality moves policy success more reliably than architecture iteration at this stage of the field.

What We Would Do Differently

Every program teaches something, and this one taught us three corrections we have since made standard. First, we would start the operator certification cohort a full week before station bring-up finishes, not after. Waiting for hardware serialized two ramps that could have run in parallel, and it cost the program most of week one’s output. Second, we would pilot every task card ourselves before handing it to operators. Two of the 40 cards contained ambiguous success criteria (“lid secured” without a torque or rotation definition), and the resulting label inconsistency did not surface until the week-five training run flagged suspicious loss on exactly those tasks. A ten-minute pilot per card would have caught both. Third, we would set the deformable-object tasks to a slower commanded pace from day one. The week-six discovery that pace, not hardware, drove the jar-opening rejection rate had been sitting in the per-task rejection data since week two; we were looking at floor-level averages and missed it. Per-task dashboards are now default on every program we run.

None of these are exotic lessons. That is rather the point: scaling demonstration data is a discipline of small, boring corrections applied early, and the gap between 1,900 episodes and 12,600 episodes was made of exactly such corrections.

The Reusable Checklist

Scaling teleoperation collection comes down to seven decisions made early:

Match rig to task family (leader-follower for fine bimanual work; VR for mobile/humanoid tasks)
Dedicated, certified operators, never rotating volunteers
A throughput gate before production collection
Written task cards with randomization zones and success criteria
Hardware timestamps on every stream; sync checked at ingest, not at training
Nightly automated QA with per-operator feedback loops
A rejection buffer of at least 15 percent in the episode target

Rig options, cost-per-episode math, and QA thresholds are covered in depth in the complete teleoperation data collection guide.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

How many episodes does a VLA model program need?

Program-dependent, but the humanoid teams we support typically target 10,000 to 50,000 QA-passed episodes per quarter across dozens of tasks. Single-task ACT policies can work from about 50 clean demos; multi-task generalization is what drives volume.

How long does it take to scale a teleoperation data program?

With rigs, certified operators, and ingest QA in place, expect three to four weeks to steady state. This program reached about 1,400 usable episodes per week from 8 operators by week four.

What rejection rate should I budget for?

At least 15 percent at maturity and 25 to 30 percent during the first two weeks of any new task or rig. Budgeting zero rejection is the most common planning error we see.

Is it cheaper to collect teleoperation data in-house or outsource it?

In-house costs more than teams expect once engineer time is counted; the client here spent eight engineer-weeks producing data that mostly failed QA. Outsourcing wins on time-to-data in most cases; in-house wins when collection is itself a core competency you intend to build.

What improved policy success most in this case study?

Rig fidelity and QA gating together. Joint-level leader-follower control removed trajectory hesitancy, and ingest gates stopped skewed episodes from reaching training. Success rose from 41 to 68 percent with no model changes.