Cost Benchmarks Archives

Comparing Training Data Approaches for Vision-Language-Action (VLA) Models: Pros, Cons & Costs

Posted on August 4, 2026July 30, 2026 by Sainath Gupta

Zero robot action labels exist in the world’s largest manipulation video corpora. Thousands of hours of humans pouring, folding, and assembling, on YouTube and in Ego4D-scale datasets, and not one recorded motor command among them. That single number is why the VLA data sourcing debate never resolves cleanly: the cheapest, most scalable source of manipulation data is missing the one field an action model trains on, and every source that has the field costs real money per hour.

So the budget conversation inside most teams collapses into familiar camps. Someone wants to teleoperate everything because actions are the labels. Someone wants to generate everything in simulation because it is cheap. Someone read the GR00T paper and wants to pretrain on YouTube. All three are right about the strengths and quiet about the weaknesses, and the budget only covers one of them properly.

Our thesis, argued below with published results and our own collection benchmarks: these approaches are not substitutes, they are layers, with different label completeness, different transfer properties, and per-hour costs that differ by two orders of magnitude. Treating them as interchangeable is how teams end up with 40,000 synthetic episodes that fine-tune into a policy that cannot close a real gripper on a real deformable object.

This post compares the four practical sourcing approaches: teleoperation, human video, synthetic generation, and open datasets. For each you get what it is, where it wins, where it fails, and what it costs, ending with the decision matrix we use when scoping client collections. Costs marked as DexSet benchmarks are from our own production pipeline; the rest cite primary sources.

Key Takeaways – Teleoperation is the only approach that produces complete observation-action-language triples; it costs $28-60 per robot-hour in our benchmarks. – Human video is the cheapest per hour and the most scalable, but has no action labels; it works as a pretraining layer, never as the whole diet. – Synthetic data has near-zero marginal cost and perfect labels, but pays a sim-to-real tax on contact-rich tasks. – Open datasets (Open X-Embodiment, DROID, BridgeData V2) are effectively free and belong in every mixture; their limit is embodiment and domain mismatch. – The winning pattern in published systems is a pyramid: open + web data at the base, synthetic in the middle, embodiment-matched teleoperation on top.

Approach 1: Teleoperation

Teleoperation data is collected by a human operator directly controlling the robot while every observation and command is recorded, making the robot’s own action stream the training label. This is the approach behind ALOHA/ACT (arxiv.org/abs/2304.13705), DROID (arxiv.org/abs/2403.12945), and the top layer of essentially every serious VLA pipeline including pi-zero’s roughly 10,000-hour corpus (arxiv.org/abs/2410.24164).

Pros: complete and exact action labels; embodiment-matched by construction; supports contact-rich and bimanual tasks; language can be attached at collection time while context is fresh.

Cons: real-time throughput ceiling (20-40 usable tabletop episodes per hour on our rigs, 8-15 for long-horizon work); operator skill variance shows up in the data; rigs cost real money (an ALOHA-class bimanual station is around $20k before cameras).

Cost (DexSet benchmark): $28-38 per robot-hour for VR-rig tabletop collection, $40-60 for leader-follower bimanual work, plus $8-15 per data-hour for language annotation and $5-9 for QA.

Approach 2: Human Egocentric Video

Human video data is footage of people performing tasks, captured from head-mounted or fixed cameras, containing rich visual and behavioral signal but no robot action labels. Ego4D-scale corpora and in-house egocentric capture both fall here.

Pros: an order of magnitude cheaper per hour than teleoperation; humans perform tasks at natural speed with natural strategies; enormous diversity is reachable; head-mounted capture rigs are cheap to scale across many wearers.

Cons: no action labels, so it cannot train the action head directly; embodiment gap between human hands and robot grippers; requires retargeting or representation-learning machinery to become useful.

Cost (DexSet benchmark): $15-25 per hour for structured egocentric capture with task scripts and QA, roughly half the cost of teleoperation for the same wall-clock hour.

Where it fits: as pretraining and co-training signal. NVIDIA’s GR00T N1 explicitly structures this as the wide base of its data pyramid (arxiv.org/abs/2503.14734).

Approach 3: Synthetic and Simulation Data

Synthetic data is generated in physics simulators or by generative models, with actions and annotations produced programmatically at near-zero marginal cost. Perfect labels, infinite resets, no hardware wear.

Pros: marginal cost approaches zero after environment setup; perfect ground truth including states no sensor can capture; safe exploration of failure states; trivially parallelizable.

Cons: the sim-to-real gap is worst exactly where manipulation is hardest: contact dynamics, deformables, friction, specular and transparent objects. Environment engineering is a real up-front cost that budget lines often forget. Policies trained purely in sim routinely need real-data fine-tuning anyway.

Cost: environment and asset engineering is the dominant term (weeks of engineer time per task family); generation itself is compute-priced. Treat it as high fixed cost, near-zero marginal cost.

Approach 4: Open Datasets

Open datasets are published robot demonstration corpora, led by Open X-Embodiment with over one million trajectories across 22 embodiments (arxiv.org/abs/2310.08864), DROID with 76k episodes across 564 scenes, and BridgeData V2 with roughly 60k trajectories (arxiv.org/abs/2308.12952).

Pros: free; instantly available; proven at scale (OpenVLA pretrained on ~970k OXE episodes, arxiv.org/abs/2406.09246); increasingly standardized via LeRobot formats.

Cons: your embodiment, cameras, and tasks are probably not in there; instruction quality varies wildly between constituent datasets; license terms need actual reading; everyone else trains on the same data, so it confers no advantage by itself.

Side-by-Side Comparison

Dimension	Teleoperation	Human Video	Synthetic	Open Datasets
Action labels	Complete, exact	None	Perfect, programmatic	Complete, variable quality
Cost per data-hour	$28-60 + annotation (DexSet benchmark)	$15-25 (DexSet benchmark)	Near-zero marginal, high fixed	Free
Embodiment match	Exact	Poor (human)	Configurable	Rarely exact
Contact-rich fidelity	High	Visual only	Weak to moderate	High where present
Scale ceiling	Low (real time)	High	Very high	Fixed (~1M episodes)
Language quality	Controllable at source	Must be added	Templated by default	Inconsistent
Best role	Fine-tuning top layer	Pretraining base	Mid-layer augmentation	Pretraining base

The Decision Matrix We Actually Use

The right mixture follows from two questions: how far is your embodiment from the open corpora, and how contact-rich are your target tasks?

Standard single arm, tabletop tasks: open datasets + a thin teleop fine-tuning layer (200-500 hours). Cheapest viable path.
Standard arm, contact-rich or precision tasks: add stereo teleop collection on the target tasks; synthetic helps for coverage but will not close the contact gap alone.
Dexterous hands or humanoids: cross-embodiment transfer is weakest here. Budget for substantial embodiment-matched teleoperation (1,000+ hours) over a human-video and open-data base.
Long-horizon mobile manipulation: prioritize segment-level language labels; Mobile ALOHA showed strong results co-training with roughly 50 demos per task on a broad base (arxiv.org/abs/2401.02117), so spread hours across many tasks.

Two budgeting notes that surprise first-time buyers. First, annotation and QA scale with data-hours, not with approach, so a cheap collection method with poor labels often costs more per useful training hour than an expensive one delivered clean; a $30 teleop hour that ships with diverse language labels and verified success flags beats a $15 hour that needs a relabeling pass and a 20% cull. Second, the approaches interact: a few hundred hours of well-instrumented teleoperation is also the calibration set that makes your synthetic pipeline honest, because it gives you real contact dynamics to validate against. Teams that sequence it that way (teleop first, sim scaled second) report far less sim-to-real disappointment than teams that generate first and verify never.

The pattern across every row: nobody should pay collection rates for the base of the pyramid, and nobody should expect the free layers to cover the top. For the full scoping framework, mixture ratios, and our vendor RFP scorecard, see the complete guide to VLA training data.

Next Step

if you are weighing these approaches for a real budget, read the complete VLA training data guide or book a demo and we will pressure-test your planned mixture against our collection benchmarks.

Frequently Asked Questions

What is the cheapest way to get training data for a VLA model?

Start with free open datasets (Open X-Embodiment, DROID, BridgeData V2) for pretraining, then buy only embodiment-matched teleoperation for fine-tuning. In our benchmarks that fine-tuning layer costs $28-60 per robot-hour plus annotation, and 200-500 hours is often enough for standard manipulators.

Is synthetic data good enough to train a VLA model?

Not alone. Synthetic data provides cheap coverage and perfect labels but carries a sim-to-real gap that is worst on contact dynamics and deformable objects. Published systems use it as a middle layer between web-scale video and real teleoperation.

How does teleoperation data compare to human video on cost?

In DexSet’s pipeline, structured egocentric human video runs $15-25 per hour versus $28-60 per robot-hour for teleoperation. The difference is labels: teleoperation records executable robot actions, human video does not, so they occupy different layers of the training mixture rather than competing.

Should I collect my own data or use Open X-Embodiment?

Both. OXE-scale open data is the standard pretraining base and costs nothing. Custom collection earns its price where open data is thin: your specific embodiment, your camera geometry, your tasks, and instruction phrasing diverse enough to make language conditioning work.

Why Training Data for Vision-Language-Action (VLA) Models Is the Biggest Bottleneck in Physical AI

Posted on August 3, 2026July 30, 2026 by Enoch Pakanati

Somewhere on every VLA data plan sits a line that reads “annotation and QA,” and it is the checklist item nearly everyone skips when the budget gets real. Collection hours get scoped to the dollar. GPU spend gets scoped to the node. The layers that decide whether those hours train anything get a checkbox. That skipped line is usually where we find the explanation when a team arrives with a Prismatic-style backbone, sensible action head, clean training code, and a model that still cannot pick up a mug it has never seen when the instruction is phrased slightly differently.

The deeper reason is structural, not a bug. Language models got to scrape a web that humanity spent thirty years writing. VLA models need trajectories where a physical robot moved and every action was recorded, and that data did not exist until someone paid to create it, one teleoperated episode at a time. Open X-Embodiment, the largest pooled open corpus, holds just over one million trajectories from 22 embodiments (arxiv.org/abs/2310.08864). Web-scale image-text corpora hold billions of pairs. Physical AI is trying to reach foundation-model behavior on roughly one-thousandth of the data its sibling fields consider table stakes.

This post argues one thesis: the VLA data bottleneck is not a single constraint but a four-layer stack (collection throughput, action-label fidelity, language annotation, QA), and teams that budget only for the first layer buy datasets that train worse models than their size suggests. We show where each layer binds, what it costs to push against, and which strategies the leading labs use to spend around it efficiently. We collect and QA this data for a living at DexSet, so the cost numbers are our own production benchmarks, flagged as such.

Key Takeaways – The robot data gap is around three orders of magnitude versus web vision-language corpora, and it cannot be closed by scraping. – The bottleneck has four layers: collection throughput, action-label fidelity, language annotation quality, and QA. Most teams only budget for the first. – RT-1 took 17 months and 13 robots to produce ~130k episodes; that throughput ceiling is why teleoperation economics matter. – In our benchmarks, usable teleop data costs $28-60 per robot-hour to collect, plus $8-15 per hour for language annotation and $5-9 for QA. – The efficient escape routes are cross-embodiment pretraining, web co-training, and spending fresh budget on diversity rather than raw episode count.

What Makes VLA Training Data a Bottleneck?

The VLA data bottleneck is the mismatch between how fast robot demonstration data can be produced and how much of it foundation-model training consumes. Every VLA needs episodes containing synchronized observations, executed actions, and language instructions, and each of those three elements has its own production constraint.

Consider the throughput math. RT-1’s dataset took a fleet of 13 robots 17 months to reach roughly 130k episodes across about 700 tasks (arxiv.org/abs/2212.06817). That is real-world, well-funded, industrial-scale collection, and it averages out to under 20 usable episodes per robot per day once resets, failures, and curation are counted. On our own rigs we see 20-40 usable tabletop episodes per collection hour on a good day, and 8-15 for long-horizon bimanual work. There is no caching layer for the physical world. Every episode costs wall-clock time on hardware.

The Four Layers of the Bottleneck

The bottleneck is a stack of four constraints, and underestimating any one of them produces a dataset that trains a worse model than its size suggests.

1. Collection throughput. Robots and teleoperators produce data in real time. A 10,000-hour corpus, the scale pi-zero reports pretraining on (arxiv.org/abs/2410.24164), is five years of single-rig operation or a coordinated fleet effort. Throughput is bought with parallel stations, trained operators, and fast reset design, all of which show up in cost per hour.

2. Action-label fidelity. Actions are the labels, and they are only as good as your sync. Continuous-action models (flow matching, diffusion heads) train on action chunks, and timestamp skew between frames and commands degrades them measurably. We hold frame-to-action skew under 10 ms on our rigs because our fine-tune ablations show loss curves separating beyond that.

3. Language annotation quality. An episode without a good instruction is imitation data, not VLA data. Templated labels teach a dozen verbs; natural, diverse instructions teach language conditioning. This layer adds $8-15 per data-hour in our pipeline, and skipping it is the single most common false economy we see.

4. QA and success verification. Failed episodes, teleoperator pauses, dropped frames, and mislabeled successes all pollute training. A 5-10% contamination rate is enough to flatten fine-tune gains on precision tasks, which is why QA is a priced line item and not a virtue.

Why You Cannot Scrape Your Way Out

Web video contains no action labels, so it cannot substitute for the action-labeled core of VLA training. Human video (Ego4D-scale corpora, YouTube) shows what manipulation looks like but never records the motor commands that produced it. The field’s answer, formalized in NVIDIA’s GR00T N1 data pyramid (arxiv.org/abs/2503.14734), is to use web and human video as a broad lower layer for visual priors, synthetic data in the middle, and real teleoperation as the small, expensive, indispensable top layer. The pyramid reduces how much teleop you need. It does not remove the layer.

What the Bottleneck Costs, in Numbers

The cost of VLA training data is best expressed per robot-hour with annotation and QA broken out, because bundled per-episode pricing hides where the money goes. Our production benchmarks:

Layer	Typical Cost	Failure Mode If Skipped
Teleop collection (tabletop, VR rig)	$28-38 per robot-hour	None; this is the floor
Teleop collection (bimanual, leader-follower)	$40-60 per robot-hour	Under-scoped dexterity data
Language annotation (episode-level)	$8-10 per data-hour	Policy ignores instructions
Language annotation (segment-level)	$12-15 per data-hour	No long-horizon capability
QA + success labels	$5-9 per data-hour	5-10% contamination, flattened gains

A 2,000-hour fine-tuning corpus, fully annotated and QA’d, lands between roughly $110k and $170k. That number stops feeling large when compared against the GPU spend it protects: a fine-tune run on contaminated or label-poor data burns the same compute and returns a weaker policy.

How Leading Teams Route Around It

The efficient strategies all share one principle: never pay to collect what transfer can give you, and spend fresh budget on what transfer cannot.

Start from cross-embodiment pretraining. OpenVLA’s 7B model, pretrained on ~970k OXE episodes, outperformed the 55B RT-2-X on its 29-task evaluation (arxiv.org/abs/2406.09246). The base layer is a solved purchase; open weights exist.
Co-train with web vision-language data. RT-2 showed this preserves open-vocabulary semantics (arxiv.org/abs/2307.15818). It costs mixture engineering, not collection dollars.
Buy diversity, not depth. Past roughly 50-150 demonstrations per task, marginal episodes decay in value; Mobile ALOHA reached strong task performance co-training with about 50 demos per task (arxiv.org/abs/2401.02117). New tasks, scenes, objects, and phrasings beat episode 400 of a solved task.
Match embodiment where it counts. Cross-embodiment transfer is weakest into dexterous hands and humanoids, so reserve embodiment-matched collection for exactly those gaps.

One more route deserves a mention because it is quietly effective: instruction relabeling. DROID and several lab pipelines have used vision-language models to generate or augment instructions on already-collected episodes, which raises language diversity on data you have already paid for. It does not fix bad sync or unlabeled failures, and machine-generated labels need a human QA sample, but as a cost lever it is hard to beat: you are improving the most neglected layer of the stack at annotation prices rather than collection prices.

The common thread in all five strategies is that they treat the bottleneck as a portfolio problem. No single source escapes it. A deliberate stack of open data, web co-training, synthetic coverage, relabeling, and targeted teleoperation gets a team to a working policy at a fraction of the naive all-teleop cost, and every published frontier system is some version of that stack.

We cover the full scoping framework, including the RFP scorecard we make vendors fill out, in our complete guide to VLA training data.

Next Step

if you want to see what clean, sync-verified, language-diverse teleop data looks like before budgeting a collection, download our free LeRobot-format sample pack, or read the full guide first.

Frequently Asked Questions

Why is training data the bottleneck for VLA models instead of compute?

Compute is fungible and rentable; robot demonstrations are produced in real time on physical hardware. The largest open robot corpus holds about one million trajectories, roughly one-thousandth of web vision-language scale, and every additional hour must be physically collected.

How do I scale training data for VLA models on a startup budget?

Pretrain or start from open cross-embodiment checkpoints (OpenVLA, Octo), co-train with web data to keep semantics, and spend collection budget only on embodiment-matched, task-matched fine-tuning data with diverse language labels. Expect $110k-170k for a fully annotated 2,000-hour corpus at current market rates.

Can simulation solve the VLA data bottleneck?

Simulation reduces but does not remove real-data needs. Sim-to-real gaps in contact dynamics and visual realism persist, which is why current pipelines (including GR00T N1) use synthetic data as a middle layer under a real teleoperation layer, not as a replacement.

How many demonstrations per task does a VLA fine-tune need?

The imitation-learning literature repeatedly lands around 50-150 demonstrations per task when fine-tuning on top of a broad pretrained base, with more needed for contact-rich or long-horizon tasks. Diversity across tasks and phrasings matters more than pushing any one task far past that floor.

The Complete Guide to Training Data for Vision-Language-Action (VLA) Models (2026)

Posted on July 31, 2026July 30, 2026 by Sainath Gupta

Five billion image-text pairs sit in LAION-5B (arxiv.org/abs/2210.08402); just over one million robot trajectories sit in Open X-Embodiment, the largest open robot dataset ever assembled (arxiv.org/abs/2310.08864). Those two numbers should not both describe the training diet of a single model family, yet every production VLA pairs a vision-language half fed at web scale with an action half starved at roughly one-thousandth of it. That mismatch, more than any architecture choice, is the defining constraint of physical AI in 2026.

The gap exists because the two halves come from different worlds. The vision-language half was pretrained on image-text pairs that already existed. The action half needs robot trajectories that do not exist until someone physically collects them, frame-synchronized, action-labeled, and paired with language instructions. You cannot scrape your way out of that. Which is why, on most VLA programs, the architecture question gets settled in a week and the compute budget in a month, while the question of how many language-annotated trajectories the model needs, at what task diversity, on which embodiments, goes unanswered the longest.

This guide gives you the full picture: what VLA training data is, how the landmark models (RT-1, RT-2, OpenVLA, Octo, pi-zero, GR00T) actually sourced their data, how action representation changes your labeling requirements, what collection and annotation cost per hour, and how to scope a dataset that transfers to your embodiment instead of overfitting to someone else’s lab. The thesis we argue throughout, and the one our delivery data keeps confirming: once you stand on the open-data base layer, dataset composition (task diversity, instruction diversity, capture fidelity) predicts VLA capability better than episode count, so budget should follow distribution, not volume.

DexSet supplies egocentric, exocentric, teleoperation, mono, and stereo data to robot foundation model teams. The numbers in this guide come from our own rigs, QA pipelines, and cost benchmarks, alongside the primary papers. Where a figure is our internal benchmark rather than a published result, we say so.

TL;DR: Key Takeaways – VLA training data consists of robot trajectories (synchronized camera frames + robot actions) paired with natural-language instructions, usually collected via teleoperation. – Landmark scale points: RT-1 used ~130k episodes over 17 months; OpenVLA pretrained on ~970k episodes from Open X-Embodiment; pi-zero reports on the order of 10,000 hours of dexterous robot data. – Action representation drives labeling spec: discrete action tokens (RT-2, OpenVLA) tolerate lower control rates; continuous-action heads (pi-zero flow matching, Octo diffusion) reward 50 Hz, low-jitter capture. – In our benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity; a proper language annotation pass adds $8-15 per hour of data. – Data mixture beats raw volume: co-training on web-scale vision-language data plus a well-balanced robot mixture is what gives VLAs semantic generalization. – Task and scene diversity predict generalization better than episode count once you pass a per-task floor (roughly 50-150 demonstrations per task in the imitation-learning literature).

What Is Training Data for Vision-Language-Action (VLA) Models?

Training data for VLA models is a corpus of robot demonstration episodes in which each timestep pairs visual observations, robot proprioceptive state, and executed actions, and each episode (or segment) carries a natural-language instruction describing the task. A VLA model consumes images and an instruction as input and produces robot actions as output, so its training data must contain all three aligned in time.

A single well-formed episode contains:

Observations. One or more camera streams (wrist-mounted, over-the-shoulder exocentric, or head-mounted egocentric), typically 224×224 to 1080p RGB at 10-60 fps, optionally stereo pairs or depth.
Proprioception. Joint positions, gripper state, end-effector pose, usually at the control rate.
Actions. The command stream that produced the motion: delta end-effector poses, joint targets, or gripper commands, timestamped against frames.
Language. An instruction such as “put the red mug on the top shelf,” attached at episode level, and increasingly at sub-episode segment level for long-horizon tasks.

The entity chain matters for anyone mapping this space: teleoperation rigs (ALOHA-style leader-follower arms, VR controllers, exoskeletons) produce demonstrations; demonstrations feed imitation learning; imitation learning at scale, on top of a pretrained vision-language backbone, is what turns a VLM into a VLA. The Hugging Face LeRobot project has become the de facto open format for storing these episodes, and its dataset cards are a good reference for what a clean schema looks like.

How the Landmark VLA Models Sourced Their Data

Every major VLA result is downstream of a specific, documented data strategy, and the strategies differ more than the architectures do. Reading the data sections of these papers teaches you more about building a VLA than reading the model sections.

Model (Year)	Org	Params	Robot Data	Language Source	Action Representation
RT-1 (2022)	Google	35M	~130k episodes, 13 robots, 17 months, ~700 tasks	Templated instructions	Discrete tokens, 256 bins (arxiv.org/abs/2212.06817)
RT-2 (2023)	Google DeepMind	12B / 55B	RT-1 data, co-fine-tuned with web-scale VLM data	Templated + web text	Actions as text tokens (arxiv.org/abs/2307.15818)
Octo (2024)	UC Berkeley et al.	27M / 93M	800k trajectories from Open X-Embodiment	Mixed, dataset-dependent	Diffusion head, continuous (arxiv.org/abs/2405.12213)
OpenVLA (2024)	Stanford et al.	7B	970k episodes from Open X-Embodiment	Mixed OXE instructions	Discrete tokens (arxiv.org/abs/2406.09246)
pi-zero (2024)	Physical Intelligence	~3B	On the order of 10k hours, 7 platforms, 68 task families	Natural + segment-level relabeling	Flow matching, 50 Hz continuous (arxiv.org/abs/2410.24164)
GR00T N1 (2025)	NVIDIA	2B	Data pyramid: web video + synthetic + real robot data	Mixed human + generated	Continuous, diffusion-based (arxiv.org/abs/2503.14734)

Three lessons sit in that table. First, scale climbed from 130k episodes to roughly a million in two years, but the biggest capability jumps (RT-2 over RT-1, pi-zero over Octo-class models) came from mixture changes, not just volume. Second, OpenVLA at 7B outperformed the 55B RT-2-X on the paper’s 29-task evaluation suite, which tells you curated cross-embodiment data can substitute for an order of magnitude of parameters. Third, nobody serious trains on robot data alone anymore; co-training against web-scale vision-language corpora is standard because it is what keeps semantic knowledge (what “the leftmost ripe banana” means) alive while the model learns motor control.

Core Data Modalities for VLA Training

Modality choice determines what your VLA can learn and what your collection rig must capture. The main axes are viewpoint (egocentric vs exocentric), depth strategy (mono vs stereo), and collection method (teleoperation vs human video).

Egocentric vs Exocentric Views

Egocentric data is captured from the robot’s or demonstrator’s own viewpoint; exocentric data is captured from external cameras watching the workspace. Wrist and head cameras give the policy view-invariant, occlusion-realistic inputs and tend to transfer better across scene layouts. Exocentric views give annotators and QA reviewers the context to verify task success and label segments. In our pipelines we treat exocentric streams as partly a QA instrument: an episode with only a wrist view is roughly twice as expensive to verify because reviewers must scrub proprioception logs to confirm success.

Mono vs Stereo

Mono capture records a single RGB stream per viewpoint; stereo capture records calibrated pairs that support depth estimation. Most published VLAs train on mono RGB because the backbone VLMs are mono. Stereo earns its extra cost on contact-rich and clutter-heavy manipulation, where depth ambiguity is a dominant failure mode. Our rule of thumb: mono for pretraining breadth, stereo for the fine-tuning set on your target tasks.

Teleoperation vs Human Video

Teleoperation produces the gold standard because the action stream is the robot’s own. Leader-follower rigs like ALOHA (arxiv.org/abs/2304.13705) hit high dexterity at a hardware cost around $20k per station; VR-based rigs are cheaper and faster to train operators on but add pose-retargeting noise. Human egocentric video (Ego4D, EgoExo4D scale) has no action labels, so it enters VLA training as a co-training or pretraining signal, the approach GR00T N1 formalizes as a data pyramid. Human video buys you visual and behavioral priors cheaply; it never replaces the teleop layer where actions live.

Action Representation: Tokenization vs Continuous Actions

Action representation is the choice of how robot actions are encoded for the model to predict, and it silently rewrites your data spec. There are two dominant families.

Discrete action tokenization (RT-1, RT-2, OpenVLA) bins each action dimension, typically into 256 buckets, and treats action prediction as next-token prediction. It plugs directly into a language-model backbone and tolerates modest control rates. The FAST tokenizer (arxiv.org/abs/2501.09747) pushed this further with a compression-based scheme that makes autoregressive VLAs viable on high-frequency data.

Continuous action heads (Octo’s diffusion head, pi-zero’s flow matching) regress action chunks directly. They excel at dexterous, high-rate control (pi-zero runs 50 Hz) but they punish sloppy capture. Timestamp jitter between frames and actions that a tokenized model shrugs off will smear a flow-matching target distribution.

What this means for data buyers, from our QA logs:

For tokenized targets, 10-15 Hz aligned capture is usually sufficient; invest the savings in task diversity.
For continuous heads, capture at 30-60 Hz with hardware-timestamped sync; we hold frame-to-action skew under 10 ms on our teleop rigs because beyond that we can measure it in fine-tune loss.
Action-chunk prediction (introduced with ACT on ALOHA) means episodes must be free of teleoperator pauses mid-chunk, which is a QA check most teams add only after their first bad fine-tune.

Language Annotation: The Layer Everyone Underestimates

Language annotation is the process of attaching instructions to robot episodes, and it is where VLA datasets most often quietly fail. A VLA is only as language-conditioned as its labels are diverse.

Templated vs natural instructions. RT-1-era datasets used templates (“pick X”, “move X near Y”). Templates are cheap and consistent, and they teach the model a grammar of about a dozen verbs. Models trained on templated instructions follow paraphrases poorly; the semantic generalization RT-2 demonstrated came from the web co-training, not the robot labels. Current practice, visible in the pi-zero and DROID (arxiv.org/abs/2403.12945) pipelines, is natural free-form instructions plus paraphrase augmentation.

Episode-level vs segment-level labeling. Episode-level labeling attaches one instruction to a whole demonstration. Segment-level labeling breaks a long-horizon episode into sub-tasks (“open the drawer”, “place the spoon inside”, “close the drawer”) with aligned timestamps. Segment labels cost more but unlock long-horizon training and better credit assignment. In our pipeline, segment-level labeling costs roughly 2.5x the episode-level pass because annotators must scrub video and mark boundaries, not just watch and describe.

Our first-hand benchmark: a language annotation pass adds $8-15 per hour of robot data depending on instruction density and whether we run single or dual annotation. We also run an instruction diversity QA gate: across any 1,000-episode batch we require minimum thresholds for unique verbs, unique object references, and spatial-relation phrases, because we have watched fine-tunes on label-poor batches lose the ability to follow anything but the majority phrasing.

Data Mixtures, Co-Training, and Cross-Embodiment Transfer

A data mixture is the weighted combination of datasets a VLA trains on, and mixture design is now a first-class engineering discipline. Three decisions dominate.

Web co-training ratio. RT-2 established that co-fine-tuning on web vision-language data alongside robot data preserves semantic breadth. Teams that fine-tune on robot data alone consistently report the model “forgetting” open-vocabulary grounding.

Cross-embodiment weighting. Open X-Embodiment showed that pooling data across 22 robot types improves average success, and OpenVLA and Octo both hand-tuned per-dataset mixture weights rather than sampling uniformly. Transfer across embodiments is real but not free: it appears strongest between similar morphologies (single-arm manipulators with parallel grippers) and weakest into dexterous hands and humanoids, which is exactly why humanoid programs still commission embodiment-matched collection.

There is no published universal mixture ratio, and be suspicious of anyone who quotes one. What the papers do agree on is the shape of the curve: heavily downweight any single dataset that would dominate by timestep count, keep a meaningful web co-training fraction throughout fine-tuning rather than dropping it to zero, and upweight the small datasets that match your target embodiment and camera geometry. In practice we see teams land between 10% and 40% embodiment-matched data by timestep in their final fine-tuning mixtures, with the exact point set by held-out evaluation rather than by principle. Measure mixtures in timesteps or tokens, not episodes; episode lengths vary enough between sources that episode-level accounting quietly misstates composition by 2-3x.

Diversity vs depth. Once a task passes a demonstration floor (Mobile ALOHA showed strong results co-training with as few as 50 demonstrations per task on top of a broad base, arxiv.org/abs/2401.02117), the marginal episode of task 12 is worth less than the first episode of task 13. When we rebalance client collections, we usually shift budget from episode count toward scene, object, and instruction diversity, and the downstream evaluations reward it.

Cost and Economics: What VLA Training Data Actually Costs

VLA data cost is driven by four stacked line items: rig amortization, operator time, annotation, and QA. Vendors who quote a single opaque per-episode price are hiding this stack. Here is ours, from DexSet production benchmarks (round figures, typical ranges we see):

Cost Component	Typical Range	Notes
Teleop collection (VR rig, tabletop tasks)	$28-38 per robot-hour	Includes operator, station amortization, resets
Teleop collection (leader-follower, bimanual/contact-rich)	$40-60 per robot-hour	ALOHA-class rigs, slower resets, higher operator skill
Language annotation, episode-level	$8-10 per data-hour	Natural instructions, single pass
Language annotation, segment-level + paraphrases	$12-15 per data-hour	Timestamped sub-task boundaries, dual pass on 10%
QA and success verification	$5-9 per data-hour	Success labels, sync checks, instruction diversity gate
Useful episodes per collection hour	20-40 tabletop; 8-15 long-horizon	After discarding failed/aborted episodes

Worked example: a 2,000-hour bimanual manipulation corpus with segment-level language and full QA lands between $114k and $168k. Against that, note that pi-zero-class results were built on roughly 10,000 hours. This is why mixture design and cross-embodiment pretraining matter commercially: nobody fine-tuning a foundation model should be paying to recollect the base layer that OXE-scale open data already provides. Pay for the embodiment-matched, task-matched, well-labeled top of the pyramid.

Case Study Proof: Scaling a Fine-Tuning Corpus for a Humanoid Team

A humanoid foundation model team came to us with an OpenVLA-style base model and a fine-tuning problem: strong tabletop scores, weak performance on their bimanual kitchen tasks, and instructions that only worked when phrased like the training templates. We scoped a 1,400-hour teleoperation collection across 42 task families with a hard instruction-diversity requirement (no template allowed to exceed 8% of a batch) and segment-level labels on all long-horizon episodes.

The mixture, not the volume, did the work. Their prior corpus had 3x our episode count but 9 task families and 14 instruction templates. After fine-tuning on the new corpus blended 60/40 with their legacy data, their internal evaluation showed task success on held-out kitchen tasks roughly doubling, and paraphrased-instruction success moved from near-random to parity with template phrasing. We report those as directional results from the client’s own evaluation harness, not audited benchmarks. The full breakdown is in our case study write-up.

Scoping Your Own Collection: The RFP Scorecard

A VLA data RFP scorecard is a structured checklist for comparing data vendors on the dimensions that predict fine-tune quality. If you take one action from this guide, make vendors answer these in writing:

Frame-to-action sync tolerance, in milliseconds, with measurement method.
Control and capture rates, and whether timestamps are hardware or software.
Instruction sourcing: templated, natural, or paraphrase-augmented, with a diversity metric per batch.
Episode-level vs segment-level labeling, priced separately.
Success-labeling protocol and inter-annotator agreement.
Cost per robot-hour, broken into collection, annotation, and QA. Refuse bundled per-episode pricing.
Delivery format (LeRobot-compatible schemas save your team weeks).

We publish ours as a downloadable template. [Download the VLA Data RFP Scorecard] and use it against us too.

Related reading this week:

Next Step

If you are scoping a VLA data collection this quarter, two options. Book a demo and we will walk through our teleop rigs, annotation pipeline, and the instruction diversity QA gate live. Or download sample data: a free LeRobot-format sample pack of bimanual teleop episodes with segment-level language labels, so your team can inspect sync quality before you talk to anyone.

Frequently Asked Questions

What is training data for vision-language-action (VLA) models?

It is a corpus of robot demonstration episodes pairing synchronized camera observations and robot actions with natural-language instructions. VLA models train on it, usually via imitation learning on top of a pretrained vision-language backbone, to map images plus an instruction to robot actions.

How much robot data does a VLA model need?

Pretraining-scale efforts use hundreds of thousands of episodes (OpenVLA used ~970k from Open X-Embodiment; pi-zero reports on the order of 10,000 hours). Fine-tuning an existing VLA to a new embodiment or task set typically needs far less: the literature shows meaningful results from 50-150 demonstrations per task on top of a broad base, with 500-2,000 hours being a common commercial fine-tuning corpus.

What does VLA training data cost?

In DexSet’s benchmarks, teleoperation collection runs $28-60 per robot-hour depending on rig and task complexity, language annotation adds $8-15 per data-hour, and QA adds $5-9. A 2,000-hour fully annotated corpus typically lands between $110k and $170k.

Do I need stereo or depth data for VLA training?

Not for most pretraining, since mainstream VLA backbones consume mono RGB. Stereo or depth earns its cost on contact-rich, cluttered, or precision tasks in your fine-tuning set, where mono depth ambiguity is a leading failure mode.

Can I train a VLA on human video instead of teleoperation?

Only partially. Human egocentric video has no robot action labels, so it serves as a pretraining or co-training signal (the GR00T N1 data pyramid approach), improving visual and behavioral priors. The action-labeled teleoperation layer remains mandatory for the policy to output executable control.

What is the difference between action tokenization and continuous actions?

Tokenized VLAs (RT-2, OpenVLA) discretize each action dimension into bins and predict actions as tokens, which suits language-model backbones and moderate control rates. Continuous-action VLAs (pi-zero with flow matching, Octo with diffusion) regress action chunks directly, enabling 50 Hz dexterous control but demanding tighter timestamp sync in the training data.

Case Study: How We Scaled Data Capture Hardware for Embodied AI for a VLA Model

Posted on July 29, 2026July 24, 2026 by Enoch Pakanati

Two numbers sat in the same planning deck: 40 captured hours a week, and 8,000 hours due in five months. Both were accurate, and they could not both survive. Hitting 8,000 hours in roughly 22 weeks means averaging about 370 delivered hours a week, nine times what the floor was producing; at the actual run rate, the dataset would have arrived roughly four years late, well after the funding milestone it was supposed to unblock. This is the point where a VLA foundation model team brought the problem to us.

The gap was not effort, and that is the thesis of this case study: scaling capture is an architecture problem (sync, formats, parallel cheap seats), not an effort or premium-hardware problem. Their two-person capture team worked hard on good hardware, a single ALOHA-class station and a Vision Pro teleop pilot. The gap was architecture: no sync strategy across devices, no shared episode format, and a fleet plan that scaled capex faster than throughput. This is the most common shape of the scaling problem we see, so we are publishing the anonymized playbook.

This post walks through the three phases of the engagement: the audit, the hardware re-spec, and the scale-out, with the real throughput and cost numbers at each stage. If you are sizing a capture floor for VLA fine-tuning, the tables below are the ones we wish every team had before their first purchase order.

DexSet operates egocentric, teleoperation, and exocentric capture lines in production; this engagement ran on our floors with the client’s task list and QA criteria. Per our client agreements, the team is anonymized (“a VLA foundation model team”) and all numbers are rounded operational figures.

Key Takeaways – A VLA foundation model team needed 8,000 hours of manipulation data in five months; their in-house rate was 40 hours a week. We reached a sustained 400 hours a week by week nine. – The fix was mostly architectural: hardware sync, one episode format (LeRobot-compatible HDF5), and parallel cheap seats instead of premium single rigs. – Fleet mix at full scale: 6 GELLO-equipped arm cells, 3 ALOHA-class bimanual stations, 12 Quest 3 teleop seats, 20 UMI-style grippers, and one 6-camera D455 evaluation studio. – Delivered cost settled at $31-42 per teleop hour across task families, inside our standard $28-60 benchmark range.

Phase 1: The Audit (Weeks 1-2)

A capture audit is a measured baseline of usable hours per seat per shift, and it always starts by separating captured hours from delivered hours. The client’s 40 weekly hours were captured hours; after QA, delivered hours were closer to 26. Two findings explained most of the loss.

First, synchronization. The Vision Pro pilot streamed hand pose to the robot over Wi-Fi with no shared clock, and QA was rejecting roughly a quarter of episodes for observation-action misalignment beyond their 25 ms tolerance. Second, format debt: the ALOHA station wrote its own HDF5 layout, the Vision Pro stack wrote JSON pose logs plus video, and an engineer spent about a day a week converting between them.

Neither problem is exotic. Both are invisible until you measure delivered hours instead of captured hours.

Phase 2: The Hardware Re-Spec (Weeks 3-5)

A hardware re-spec is a redesign of the fleet around delivered cost per hour rather than device quality, and ours cut the premium hardware first. The Vision Pro seats moved to evaluation-only duty. For fleet teleop we standardized on Quest 3 headsets at roughly $500 per seat: seven times cheaper per seat, no measurable difference in demonstrations per operator-hour on the client’s pick, place, and insertion task families, and cheap enough to hold 20 percent spares.

For arm-based capture we kept the existing ALOHA-class station (fine bimanual tasks justified it) and added GELLO leader arms (arxiv.org/abs/2309.13037) at about $300 per arm on follower robots the client already owned. For task families that did not need force or proprioception fidelity, we added UMI-style handheld grippers (arxiv.org/abs/2402.10329) at roughly $400 per unit, with a budgeted SLAM QA pass.

The last addition was a six-camera RealSense D455 evaluation studio, hardware-synced through the D455’s trigger pin, wrapped around one teleop cell. Every evaluation episode was therefore captured ego and exo simultaneously, which the client’s research team used to test viewpoint generalization in the style of Ego-Exo4D (arxiv.org/abs/2311.18259). The studio cost about $2,700 in cameras and roughly the same again in truss, sync wiring, lighting, and NVMe ingest, a ratio worth budgeting for in advance.

Every line wrote one format: LeRobot-compatible episodes (github.com/huggingface/lerobot) with per-frame timestamps from a shared NTP-disciplined clock and a hardware sync signal on the camera side. The conversion engineer got their day back.

Phase 3: Scale-Out (Weeks 6-9)

Scale-out is the replication of a proven seat design in parallel, and it only works after yield per seat is stable. We added seats in weekly increments, holding the QA gate constant, until the floor hit the target run rate.

Metric	Baseline (client in-house)	Week 5 (re-spec live)	Week 9 (full scale)
Captured hours / week	40	130	~460
Delivered hours / week (post-QA)	26	110	~400
QA rejection rate	~35%	15%	10-12%
Active seats	3	14	42
Fleet capex (cumulative)	~$32k	~$68k	~$118k
Delivered cost per teleop hour	not tracked	$46	$31-42

Two numbers deserve emphasis. The rejection rate fell from 35 percent to under 12 percent before most of the new seats arrived; sync and format fixes, not hardware volume, did that. And full-scale fleet capex, about $118k for 42 seats, is less than many teams spend on four premium bimanual cells. Cheap synchronized seats, bought after the yield problem was solved, are what made the 8,000-hour target arrive on schedule.

The non-hardware work mattered too, and it is easy to undercount. Operator onboarding became a two-day curriculum with a certification task instead of an afternoon of shadowing, which pulled new-seat ramp time down from three weeks to one. The QA gate ran four automated checks (sync residual, calibration freshness, trajectory smoothness, format validity) plus human review on a 10 percent sample, and its thresholds never moved during scale-out. Holding the gate constant is what made the weekly delivered-hours number trustworthy enough to plan a training schedule against.

What We Would Do Differently

Honest retrospectives are rarer than case studies, so here is ours. We would have moved the UMI-style gripper line earlier in the schedule; its SLAM post-processing pipeline took three weeks to tune, and starting it in week 3 instead of week 6 would have added several hundred delivered hours. We also underestimated storage: the exo studio’s terabyte-per-shift output forced a mid-project NVMe retrofit that a day of arithmetic would have predicted. Neither mistake threatened the deadline. Both are now checklist items in every new engagement.

What Generalizes From This Engagement

The transferable lessons are hardware-selection rules, and they match the decision matrix in our full guide to data capture hardware for embodied AI:

Measure delivered hours, not captured hours. The 14-hour weekly gap at baseline was invisible in the client’s dashboards.
Fix sync before buying anything. A 25 ms alignment tolerance is unreachable over ad-hoc Wi-Fi streaming without a shared clock.
Premium headsets are evaluation tools, not fleet tools. Quest 3 matched Vision Pro on demonstrations per operator-hour for standard manipulation families.
Match rig fidelity to task family. ALOHA-class cells for fine bimanual work, GELLO for arm tasks on owned robots, UMI for volume, one format across all of them.
Scale last. Every seat added before yield stabilizes multiplies your rejection rate.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

How long does it take to scale a robot data capture operation?

In this engagement, nine weeks from audit to a sustained 400 delivered hours per week. The first five weeks fixed synchronization, formats, and seat design; scaling seats was the fast part.

How much did the capture hardware cost at full scale?

About $118,000 in cumulative fleet capex for 42 seats: 12 Quest 3 teleop seats, 6 GELLO arm cells, 3 ALOHA-class stations, 20 UMI-style grippers, and a 6-camera RealSense D455 evaluation studio.

What QA rejection rate is normal for teleoperation data?

On a well-synchronized floor we hold 10 to 12 percent. Rates above 25 percent almost always trace to clock misalignment or ergonomic problems, not operator error.

Did the VLA team really not need Vision Pros?

Not for fleet capture. On pick, place, and insertion tasks, ~$500 Quest 3 seats matched the $3,499 Vision Pro on demonstrations per operator-hour. The Vision Pros stayed in service for gaze-labeled evaluation episodes, where their eye tracking earns the price.

Can we run this playbook in-house instead of using a vendor?

Yes; the phases (audit, re-spec, scale-out) and rules above are the playbook. The vendor case is throughput on demand without capex and hiring, which mattered here because of the five-month deadline.

Facing a similar deadline?

[Book a Demo] and we will run the throughput math on your task list, or start with the full guide to data capture hardware for embodied AI.

Why Data Capture Hardware for Embodied AI Is the Biggest Bottleneck in Physical AI

Posted on July 27, 2026July 24, 2026 by Enoch Pakanati

There is a rule on our capture floor that surprises every visitor: no rig purchase gets approved until someone writes down, and signs, the usable hours per shift it is expected to deliver. The rule exists because of a ratio most teams meet too late. A training cluster can process a million episodes a day; a capture floor produces a few hundred. That ratio, not model architecture, is the thesis of this post: capture hardware throughput is the binding constraint in physical AI, and almost every team we work with discovers it after they have already hired the ML engineers.

The gap exists for a structural reason. Language models scaled on data that already existed; the internet was captured for free, decades in advance, by billions of people. Robot data does not exist until someone straps on a camera or sits down at a teleoperation rig and creates it, one real-time hour at a time. Open X-Embodiment, the largest cross-robot dataset effort to date, pooled contributions from more than 20 institutions to assemble on the order of a million trajectories (arxiv.org/abs/2310.08864). GPT-class text corpora are trillions of tokens. The disparity is not a funding problem. It is a hardware throughput problem.

This post breaks down where the bottleneck actually sits: capture-rate physics, sync failure, operator throughput, and fleet reliability. You will get the numbers we use to size capture floors, and a checklist for finding your own pipeline’s binding constraint.

DexSet runs egocentric, teleoperation, and multi-view exocentric capture lines in production for robotics foundation model teams, so the failure modes below are ones we have paid for personally.

Key Takeaways – Robot data must be created in real time; one hour of demonstrations costs one hour of human effort plus overhead, which makes capture hardware throughput the binding constraint in physical AI. – The bottleneck has four layers: capture-rate physics, cross-sensor synchronization, operator ergonomics, and fleet reliability. Most teams only budget for the first. – In our production benchmarks, a well-run teleop cell yields 4 to 5 usable data hours per 8-hour operator shift; a badly specified one yields under 2. – Scaling is a parallelism problem: 10 cheap synchronized seats beat 2 premium ones on delivered hours per dollar, every time we have measured it.

The Bottleneck Is Physical, Not Computational

The core constraint in embodied AI is that demonstration data is generated at the speed of physical reality, one wall-clock hour per data hour, per rig. No compiler optimization changes this. A capture floor’s output is seats multiplied by shift hours multiplied by yield, and hardware choices control two of those three factors.

Consider the arithmetic that surprised one of our clients. They wanted 10,000 hours of teleop demonstrations in six months. At a realistic 4.5 usable hours per operator shift, that is roughly 2,200 shifts, or 18 rigs running double shifts for the full period with zero downtime. Their plan had budgeted 6 rigs. The model roadmap slipped a quarter before a single GPU was misused.

Where Hardware Throughput Actually Dies

Capture throughput is lost in four places, and each maps to a hardware decision made months earlier.

1. Capture-Rate Physics

Capture-rate physics is the raw sensor and storage arithmetic of a rig: resolution times frame rate times stream count. A six-camera RealSense D455 studio recording depth plus RGB produces on the order of a terabyte per shift before compression. Teams that bought cameras without buying the matching NVMe offload path spend the last hour of every shift copying files instead of capturing. The camera was $450; the missing storage architecture costs a data hour per day, forever.

2. Synchronization Failure

Synchronization failure is misalignment between sensor clocks, and it converts captured hours into rejected hours during QA. A 40 ms drift between headset pose and robot joint states is invisible to the operator and disqualifying for action-labeled training data. Hardware with a sync story (RealSense sync pins, Project Aria’s shared-clock VRS recordings) keeps yield around 90 percent in our pipelines. Hardware without one (consumer action cams in multi-view arrays) can silently push rejection rates past 30 percent, and you find out weeks later.

3. Operator Ergonomics

Operator ergonomics is the human sustainability of a rig across a full shift, and it is the most under-priced axis in hardware selection. A 600-plus-gram headset with an external battery is fine for a 20-minute demo and brutal at hour six. On our floors, moving long-shift egocentric work from headsets to ~150 g head-mounted action cameras raised sustained weekly output per operator by a third, purely because breaks got shorter and turnover dropped. ALOHA-style seated leader arms (arxiv.org/abs/2304.13705) similarly outlast handheld free-space teleop for repetitive tabletop tasks.

4. Fleet Reliability

Fleet reliability is the fraction of seats actually capturing at any moment, and it decays with device fragility and SDK churn. Consumer devices were not designed for 60-hour weeks: straps wear out, USB ports loosen, batteries lose capacity by month four, and a firmware update can strand a whole fleet’s capture app on a Tuesday morning. Budget spare units at 15 to 20 percent of fleet size and pin firmware versions the way you pin Python dependencies. A $3,499 spare hurts, which is one more argument for $500 seats.

There is a fifth, quieter drain worth naming: calibration and QA overhead. Every worn or moved device needs its extrinsics re-verified regularly, and every captured hour needs review before it counts. On a healthy floor this overhead runs 10 to 15 percent of shift time. On a floor that skips it, the overhead appears anyway, relocated into next month’s rejection rate.

The Throughput Table

The numbers below are typical ranges from our own capture operations. Treat them as planning figures.

Capture Setup	Capex per Seat	Usable Hours per 8h Shift	Common Bottleneck	Delivered Cost Range (our benchmarks)
GoPro head-mount egocentric (~$400)	$400-700	5.5-6.5	Offload bandwidth, battery swaps	Lowest per hour of any modality
Quest 3 VR teleop (~$500 + robot)	$1,200-5,500	4-5	Passthrough latency fatigue, resets	$28-45/hr
Vision Pro VR teleop ($3,499 + robot)	$4,500-8,000	3.5-4.5	Comfort, battery, spare cost	$40-60/hr
ALOHA-class bimanual (~$20k station)	$22,000-30,000	4-5	Task resets, arm maintenance	$35-60/hr
GELLO leaders on existing arms (~$300/arm)	$600-1,000 incremental	4-5	Follower robot availability	$28-45/hr
UMI handheld grippers (~$400)	$400-600	5-6	SLAM post-processing QA	Below teleop, above raw egocentric

Read the table columns together, not in isolation. The cheapest capex row is not automatically the cheapest delivered hour once yield and QA rejection enter, but the pattern holds: parallel cheap seats with a real sync plan dominate small fleets of premium hardware.

How to Scale Past the Bottleneck

Scaling data capture hardware for embodied AI means adding synchronized parallel seats while holding per-seat yield constant, and the order of operations matters.

Instrument yield first. Measure usable hours per shift per seat before buying anything. Most teams cannot answer this today.
Fix sync before adding seats. Ten unsynced rigs scale your rejection rate, not your dataset.
Standardize on one episode format. LeRobot’s dataset format (github.com/huggingface/lerobot) or a fixed HDF5 schema; converters eat engineer-weeks.
Buy for the shift, not the demo. Weigh the device, check the battery swap time, and price spares at 15-20 percent of fleet.
Then parallelize. Cheap identical seats, one calibration procedure, one QA gate.

The order matters more than the shopping list. Teams that parallelize first and fix sync second end up owning ten rigs that produce ten streams of mutually misaligned data, and the “scaling” project quietly becomes a salvage project.

Or skip the capital expenditure entirely: this bottleneck is exactly why data vendors exist. A vendor amortizes rigs, operators, calibration routines, and QA gates across many clients, which is how delivered hours can cost less than an in-house floor’s fully loaded rate even before you count the hiring time. The build-versus-buy math is covered in our full guide to data capture hardware for embodied AI, including per-device comparison tables and a decision matrix.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

Why is data capture the bottleneck in physical AI rather than compute?

Because robot demonstrations must be created in real time by humans and hardware, one wall-clock hour per data hour per rig. Compute scales by purchase order; capture scales only by adding synchronized seats and operators.

How many usable data hours does one teleoperation rig produce per day?

In our production benchmarks, a well-specified teleop cell yields 4 to 5 usable hours per 8-hour shift after resets, calibration, and QA rejection. Poorly synchronized or ergonomically bad rigs can drop below 2.

What is the fastest way to scale robot data collection?

Fix synchronization and episode formats first, then add cheap parallel seats. Ten ~$500 synchronized capture seats reliably outdeliver two premium seats on hours per dollar.

How much data do VLA models need?

Published systems give the scale: Open X-Embodiment pooled roughly one million trajectories across institutions, and fine-tuning a VLA for a task family typically consumes hundreds to thousands of targeted demonstrations. The practical answer is more than one team’s ad-hoc rig produces, which is why capture throughput planning matters.

Does buying more expensive capture hardware increase throughput?

Not by itself. Throughput follows seats, sync, ergonomics, and reliability. A $3,499 headset that fatigues operators faster than a $500 one lowers delivered hours per dollar despite better optics.

Sizing a capture floor?

Start with the full guide to data capture hardware for embodied AI, or book a demo and we will walk you through the throughput math on your task list with real DexSet sample data.

The Complete Guide to Data Capture Hardware for Embodied AI (2026)

Posted on July 24, 2026July 24, 2026 by Sainath Gupta

A Head of Data on a scoping call last quarter asked us which single capture device her team should standardize on, and for the first time in months, we had no quick answer. We had run every device she named in production. The problem was the question: it has no device-shaped answer, because a capture device is not a purchase, it is a commitment. The hardware you choose locks your pipeline into what that device can timestamp, sync, and survive, and teams that fail on data usually failed at that moment, months before anyone touched a model. That is this guide’s thesis: capture hardware is a data-strategy decision, judged on seven axes and on cost per delivered hour, never on spec sheets.

The reason the question stumps buyers is that capture hardware for embodied AI sits in a gap between consumer electronics and lab equipment. A Quest 3 was built for gaming, not for exporting synchronized RGB, IMU, and hand pose at training quality. A RealSense was built for prototyping, not for 40 units running ten hours a day. Vendor spec sheets answer none of the questions that matter for foundation model training, so teams learn the hard lessons after the purchase order clears.

This guide gives you the full picture: every major hardware category (headsets, research glasses, action cameras, stereo depth cameras, teleoperation rigs, handheld grippers, tactile sensors), compared on the seven axes that decide whether your dataset trains a VLA model or fills a hard drive. You will leave with real prices, a cost-per-hour model, and a decision matrix mapped to the three capture strategies that matter: egocentric human data, teleoperation, and multi-view exocentric capture.

At DexSet we run these devices in production. Our teams operate teleop cells, headset fleets, and multi-camera capture studios that deliver egocentric, exocentric, mono, and stereo data to robotics foundation model teams. Every number in this guide comes from our own rigs, published research, or a linked primary source.

TL;DR: Key Takeaways – Data capture hardware for embodied AI spans four categories: wearable egocentric devices ($350 to $3,499), stereo depth cameras ($250 to $500 per unit), teleoperation rigs ($300 per arm to $20,000+ per station), and tactile sensors. – The seven evaluation axes are price, resolution/fps, depth quality, sync capability, wearability, SDK maturity, and data formats. Sync capability is the axis teams most often ignore and most often regret. – A Meta Quest 3 at ~$500 covers most teleoperation UI needs; the Apple Vision Pro at $3,499 earns its price only for eye tracking and high-resolution passthrough research. – Teleop capex is lower than most CTOs assume: a GELLO leader arm costs roughly $300 per arm, against $20,000+ for a full ALOHA-class bimanual station. – In our production benchmarks, delivered teleop data runs $28 to $60 per hour depending on rig and task complexity; egocentric human data runs meaningfully cheaper per hour but trains different capabilities.

What Is Data Capture Hardware for Embodied AI?

Data capture hardware for embodied AI is the set of physical devices, wearables, cameras, teleoperation interfaces, and sensors, used to record the visual, spatial, inertial, tactile, and action data that trains robots and vision-language-action (VLA) models. It is the acquisition layer of the robot learning stack: everything between the physical world and your training dataset.

The category matters because embodied AI models learn from demonstrations, not from text scraped off the internet. A VLA model like OpenVLA (arxiv.org/abs/2406.09246) or Physical Intelligence’s pi-0 (arxiv.org/abs/2410.24164) consumes synchronized streams of camera frames, proprioception, and actions. The hardware that records those streams sets a hard ceiling on model quality. No amount of post-processing recovers depth a sensor never measured or a timestamp a device never wrote.

Core Concepts and Data Modalities

Data modalities are the distinct sensor streams a capture device records, and every hardware decision starts by listing which modalities your model actually needs. The main ones for embodied AI are RGB video, stereo pairs, depth, IMU, eye gaze, hand and body pose, robot proprioception, and tactile signal.

Mono vs. Stereo RGB

Mono RGB is a single camera stream; stereo RGB is a calibrated pair that allows depth to be recovered by triangulation. Mono is cheaper and lighter and dominates large egocentric datasets like Ego4D (arxiv.org/abs/2110.07058). Stereo gives your model, or your post-processing pipeline, metric 3D structure, which matters for manipulation policies that must reason about distance to a grasp point. On our rigs, stereo roughly doubles storage and calibration workload, so we specify it only when the downstream model consumes it.

Depth

Depth is a per-pixel distance measurement, produced either by active sensors (structured light, time-of-flight) or by stereo matching. Active depth from a RealSense-class sensor works well indoors at 0.3 to 6 meters but degrades in sunlight and on reflective surfaces. Neural stereo depth (ZED 2i class) extends range but requires a GPU at capture time or in post.

IMU and Time Synchronization

An IMU (inertial measurement unit) records acceleration and angular velocity, typically at 200 Hz to 1,000 Hz, and it is only useful if its clock aligns with the camera clock. Cross-device sync is the silent killer of embodied AI datasets: a 40 ms offset between video and proprioception is invisible in playback and fatal for action-labeled training data. Devices differ enormously here. Project Aria timestamps all sensors on a shared clock; a GoPro gives you a self-consistent file but no native way to sync with a robot’s ROS 2 clock without an external signal.

Proprioception and Action Data

Proprioception is the robot’s own joint state (positions, velocities, torques), and action data is the command stream sent to the robot. These come from the robot or the teleoperation interface, not from cameras, which is why teleop rigs are capture hardware in their own right. Imitation learning methods like ACT, introduced with the ALOHA rig (arxiv.org/abs/2304.13705), train directly on this stream. The entity chain to keep straight: teleoperation produces demonstrations, demonstrations feed imitation learning, imitation learning trains the VLA policies that run on robots.

Tactile

Tactile sensing records contact geometry and force at the gripper, usually via vision-based sensors in the GelSight style, where a camera watches a deformable gel pad. Tactile is the newest modality in production pipelines and the least standardized; we cover the hardware below.

The Four Hardware Categories

Capture hardware for embodied AI falls into four categories: wearable egocentric devices, stereo and depth cameras for exocentric capture, teleoperation rigs, and handheld or tactile interfaces. Most production pipelines combine at least two.

Category 1: Wearable Egocentric Devices

Wearable egocentric devices are head-mounted cameras and headsets that record the world from a human’s first-person viewpoint, the same viewpoint a humanoid robot’s head camera will have. This category powers egocentric pretraining datasets like Ego-Exo4D (arxiv.org/abs/2311.18259) and doubles as the operator interface for immersive teleoperation systems like Open-TeleVision (arxiv.org/abs/2407.01512).

Device	Price	Sensors & Resolution	Depth	Sync	Wearability (8h shift)	SDK Maturity	Native Data Formats
Meta Quest 3	~$500	Color passthrough cameras, 6DoF inside-out tracking, hand tracking, IMU	Passthrough depth estimate, not export-grade	Good within headset; WebXR/OpenXR timestamps for teleop	Moderate; 515 g, front-heavy after ~2h	Strong (OpenXR, Meta SDK, large dev community)	App-level capture; pose + hand joints via OpenXR
Apple Vision Pro	$3,499	High-res micro-OLED displays, high-quality color passthrough, eye tracking, hand tracking, LiDAR	LiDAR scene depth	Excellent within device; ARKit timestamps	Poor for long shifts; 600-650 g, external battery	Strong but restrictive (visionOS, ARKit); raw camera access gated	ARKit poses, hand skeletons; limited raw sensor export
Project Aria / Aria Gen 2	Research access (not retail)	RGB camera, 2 mono SLAM cameras, eye tracking cameras, spatial mics, dual IMUs (Gen 2 adds PPG, improved compute)	No active depth; SLAM-derived	Best in class; all sensors on shared clock, VRS format	Excellent; ~75 g glasses form factor	Good (Project Aria Tools, MPS cloud services)	VRS files, MPS trajectories, gaze (projectaria.com)
GoPro head mount (Hero-class)	~$400	Single wide FOV RGB, 5.3K/60 or 4K/120, IMU telemetry	None	Weak across devices; GPMF telemetry internal only	Good; ~150 g with mount	Moderate (GPMF parsers, no live SDK)	MP4 + GPMF metadata track

The pattern to notice: the $3,499 device is not seven times better than the $500 device for data capture. Vision Pro wins when you need eye gaze and the highest-fidelity passthrough for perception research or premium teleop. Quest 3 wins on fleet economics: at our scale, one damaged Vision Pro costs the same as seven Quest 3 replacements. Aria wins on data quality per gram but is research-access only, which makes it a partnership decision rather than a purchasing decision. The GoPro remains the workhorse for raw egocentric video volume, and it is the camera inside the UMI gripper stack discussed below.

Category 2: Stereo Depth Cameras for Exocentric Capture

Stereo depth cameras are fixed or tripod-mounted sensors that record a scene from outside the actor, providing the third-person views used in multi-view exocentric datasets and in robot workspace monitoring. Datasets like DROID (arxiv.org/abs/2403.12945) standardized on this class of sensor.

Device	Price	RGB / Depth Resolution & FPS	Depth Quality	Sync Capability	SDK Maturity	Data Formats
Intel RealSense D435i	~$300-350	Depth 1280x720 @ up to 90 fps; RGB 1080p @ 30; IMU	Good 0.3-3 m; global-shutter depth sensors	Hardware sync pin across units	Mature (librealsense, ROS 2 wrapper)	ROS 2 bags, .bag/.ply, raw streams
Intel RealSense D455	~$420-450	Depth 1280x720 @ 90 fps; wider 95 mm baseline; global-shutter RGB; IMU	Better than D435i at 4-6 m range	Hardware sync pin	Mature (same stack)	Same as D435i
Stereolabs ZED 2i	~$500	Stereo 2K @ 15, 1080p @ 30, 720p @ 60; IMU, barometer, magnetometer	Neural stereo depth, usable 0.3-20 m	Software sync; multi-cam via ZED Hub	Mature but GPU-dependent (ZED SDK, needs NVIDIA)	SVO recordings, ROS 2 wrapper
Luxonis OAK-D	~$250-350	4K RGB + 2x mono stereo pair; on-device compute	Good short-range stereo depth	Hardware FSYNC on some variants	Growing (DepthAI, ROS 2 driver)	DepthAI streams, ROS 2

For multi-view exo capture, the deciding axis is sync. RealSense’s hardware sync pin lets you drive eight cameras off one trigger, which is why it still dominates academic capture cells despite Intel’s on-again-off-again commitment to the product line. ZED 2i buys you longer range and better outdoor behavior at the cost of an NVIDIA GPU per capture node. OAK-D is the value play when you want on-device neural inference to filter or annotate at capture time.

Category 3: Teleoperation Rigs

Teleoperation rigs are the leader-follower arm pairs, exoskeletons, and VR-based control systems through which a human operator produces robot action data. This is the category where hardware choice most directly sets your cost per demonstration.

System	Price	What It Is	Action Fidelity	Operator Fatigue	SDK / Ecosystem	Data Formats
ALOHA / ALOHA 2 rig	~$20,000+ per bimanual station	Two leader + two follower arms, joint-space mapping, 50 Hz	High; fine bimanual tasks	Moderate; seated, arms supported	Strong research ecosystem (ACT, Mobile ALOHA, LeRobot support)	HDF5 episodes; LeRobot dataset format
GELLO	~$300 per arm (plus follower robot)	3D-printed, Dynamixel-based kinematic-twin leader arm (arxiv.org/abs/2309.13037)	High for the matched follower	Low-moderate	Open source, LeRobot-compatible	Joint streams to ROS 2 / LeRobot
UMI gripper	~$400 build (printed gripper + GoPro)	Handheld gripper with wrist-mounted fisheye GoPro; robot-free demonstrations (arxiv.org/abs/2402.10329)	Good; SLAM-recovered actions, no proprioception	Low; natural human motion	Open source, active community	MP4 + GPMF, post-processed to episodes
VR teleop (Quest 3 / Vision Pro + arms)	$500-$3,499 + robot	Headset streams head/hand pose to robot controller (Open-TeleVision style)	Good; depends on retargeting	Moderate; passthrough latency drives fatigue	Growing; OpenXR + custom stacks	Pose streams + robot logs
Tactile (DIGIT / GelSight-style)	A few hundred dollars per fingertip (DIGIT-class); research-grade GelSight units cost thousands	Vision-based tactile sensor in gripper fingertips	Adds contact-rich signal	N/A (robot-mounted)	Moderate (open DIGIT designs; PyTouch-era tooling)	Compressed tactile image streams

The $20,000 ALOHA figure and the $300 GELLO figure describe different points on the same curve, not competing products. ALOHA is a complete bimanual station including four arms. GELLO is a leader-arm design you bolt onto follower robots you already own. UMI removes the robot from data collection entirely, which is why it produces the cheapest demonstrations and the weakest proprioception. Every serious pipeline we run mixes at least two of these.

Category 4: Multi-View Exocentric Studios

A multi-view exocentric studio is an array of four to twelve synchronized cameras surrounding a workspace, recording the same manipulation from multiple third-person angles. Ego-Exo4D paired Aria glasses with four to five stationary GoPros per scene precisely because models trained on both viewpoints generalize better than either alone. Budget $2,500 to $6,000 per studio in cameras and mounts, then double it for the sync, lighting, and storage infrastructure that spec sheets never mention.

Decision Matrix: Which Hardware for Which Use Case

The right hardware follows from the capture strategy, and there are three dominant strategies: egocentric human data, teleoperation, and multi-view exocentric capture.

Use Case	First Choice	Budget Alternative	Premium Option	Avoid
Egocentric human data at scale (pretraining)	GoPro head mounts (~$400/operator)	Used Hero-class cameras	Project Aria via research partnership	Vision Pro (cost, battery, comfort)
Egocentric with gaze + trajectories	Project Aria / Aria Gen 2	Quest 3 with logging app	Aria Gen 2	GoPro (no gaze, no shared clock)
Immersive teleoperation UI	Quest 3 (~$500)	Quest 3	Vision Pro (eye tracking, passthrough fidelity)	GoPro (not a display)
Bimanual fine manipulation demos	ALOHA-class station (~$20k)	GELLO leaders on existing arms (~$300/arm)	ALOHA 2 with tactile fingertips	UMI (no bimanual joint coupling)
Cheap single-arm demonstrations at volume	UMI grippers (~$400 each)	GELLO + one follower arm	ALOHA station	Vision Pro teleop (capex per seat)
Multi-view exo studio	4-8x RealSense D455, hardware-synced	OAK-D array	ZED 2i array with GPU nodes	Unsynced consumer webcams
Contact-rich tasks	DIGIT-class fingertips on any rig above	Single tactile finger + RGB	GelSight-style research sensors	Skipping tactile, then labeling contact by eye

Cost, Economics, and ROI

Capture hardware economics are dominated by throughput and operator time, not by device price, so the correct unit of analysis is cost per delivered data hour. This is the section competitors leave out, so here are the numbers we actually use.

Capex per seat, typical ranges:

Egocentric human capture seat: $400 to $700 (camera, mount, spare batteries, storage cards)
VR teleop seat: $1,200 to $5,500 (headset, workstation, networking; excludes the robot)
GELLO teleop seat: $600 to $1,000 on top of an existing follower arm
ALOHA-class bimanual cell: $22,000 to $30,000 fully commissioned
Exo studio (6 cameras): $2,500 to $6,000 in sensors, roughly the same again in sync, mounting, lighting, and storage

Cost per delivered hour, our production benchmarks: teleoperation data lands at $28 to $60 per hour depending on rig type, task complexity, and QA depth. Egocentric human data typically lands at one third to one half of that, because a human wearing a camera needs no robot, no retargeting stack, and far less resets-per-hour overhead. These are the typical ranges we see across our own pipelines; treat them as planning numbers, not quotes.

The ROI logic is straightforward. A $20,000 ALOHA cell running two shifts amortizes to under $2 per data hour within a year; the operators cost ten times that. Buying cheaper hardware that cuts operator throughput by 20 percent is a false economy. Buying $3,499 headsets when $500 ones sustain the same demonstrations per hour is the opposite mistake.

Case Study: A Humanoid Foundation Model Team’s Capture Stack

The proof is in a deployment, so here is an anonymized configuration from our own work. A humanoid foundation model team came to us needing three data types simultaneously: egocentric human video for pretraining, bimanual teleop demonstrations for fine-tuning, and multi-view exo recordings for evaluation.

We stood up three lines. Line one: 30 GoPro head-mount seats for kitchen and warehouse egocentric capture, delivering several hundred hours per week. Line two: four ALOHA-class stations plus six GELLO-equipped single-arm cells, producing task-labeled demonstrations in LeRobot format (github.com/huggingface/lerobot). Line three: a six-camera RealSense D455 studio, hardware-synced, wrapped around one teleop cell for paired ego-exo evaluation episodes.

Two hardware lessons generalized. First, the D455’s wider baseline halved our depth QA rejections at 2 to 4 meters compared with the D435i units we started with. Second, moving VR teleop from Vision Pro pilots to Quest 3 fleet units cut capex per seat by 85 percent with no measurable drop in demonstrations per operator-hour on pick-and-place classes of tasks. The Vision Pros stayed in service for the subset of tasks where gaze labels earned their cost.

Get the RFP Scorecard

A hardware decision this size deserves a structured evaluation, so we built one you can reuse. The DexSet Data Capture Hardware RFP Scorecard is a one-page matrix that scores any device or vendor on the seven axes in this guide (price, resolution/fps, depth quality, sync, wearability, SDK maturity, data formats) plus delivery terms and QA guarantees. Teams use it to compare quotes on equal footing instead of vendor spec sheets. Download it below, and if you want the comparison done for you, that is literally our job.

Related reading from this series:

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

What is data capture hardware for embodied AI?

Data capture hardware for embodied AI is the set of wearable cameras, headsets, depth sensors, teleoperation rigs, and tactile sensors used to record the synchronized visual, inertial, action, and contact data that trains robot policies and vision-language-action models.

Is the Apple Vision Pro worth it over the Meta Quest 3 for robotics data capture?

Only for specific needs. The Vision Pro’s eye tracking and passthrough fidelity justify its $3,499 price for gaze-labeled data and premium teleoperation research. For fleet teleop and general egocentric capture, the ~$500 Quest 3 delivers comparable demonstrations per operator-hour at one seventh the capex.

How much does a teleoperation rig cost?

Between $300 and $30,000 depending on class. A GELLO leader arm costs roughly $300 per arm on top of a follower robot you already own; a UMI handheld gripper builds for about $400; a complete ALOHA-class bimanual station runs $20,000 or more fully commissioned.

What is the difference between egocentric and exocentric data capture?

Egocentric capture records from the actor’s first-person viewpoint using head-mounted devices; exocentric capture records the actor from external, third-person cameras. Datasets like Ego-Exo4D pair both because policies trained on combined viewpoints generalize better.

Why does time synchronization matter so much in capture hardware?

Because embodied AI models train on aligned observation-action pairs. A clock offset of tens of milliseconds between camera, IMU, and robot proprioception silently corrupts action labels, and it cannot be fixed after the fact without shared timestamps or an external sync signal.

Which stereo depth camera is best for a multi-camera robotics capture studio?

For hardware-synchronized multi-view rigs, the Intel RealSense D455 (~$420-450) is the default choice because of its sync pin, 95 mm baseline, and mature ROS 2 support. The ZED 2i suits longer-range or outdoor scenes if each node has an NVIDIA GPU.

Ready to skip the hardware learning curve?

DexSet already runs the rigs in this guide at production scale. [Download the RFP Scorecard] to evaluate vendors on equal terms, or [Book a Demo] to see sample egocentric, teleop, and multi-view datasets from these exact devices. [Download Sample Data] is one click away.

Why Teleoperation Data Is the Biggest Bottleneck in Physical AI

Posted on July 20, 2026July 21, 2026 by Enoch Pakanati

In a budget review we sat in on last quarter, a founder approved a seven-figure compute reservation in about ten minutes, then spent the next hour interrogating a $70k line for demonstration data. The compute felt legible: dollars per GPU-hour, a market price. The data line had no equivalent, because nobody in the room could say what an hour of human demonstration work actually yields.

That asymmetry is the subject of this post, and the thesis is blunt: physical AI is not compute-bound, it is demo-bound. Language models trained on the internet. Vision models trained on the internet. Robot foundation models have no internet of action-labeled data, so every demonstration a VLA model consumes had to be manufactured: a person, at a rig, doing a task, one episode at a time. That production rate, not FLOPs, sets the pace of the field.

The gap exists because physical interaction data cannot be scraped. YouTube shows what hands do but not the joint torques, gripper commands, and proprioceptive states a policy needs as action labels. Simulation helps but leaks reality at every contact. That leaves teleoperation as the primary source of ground-truth action data, and teleoperation runs at human speed.

This post puts numbers on the bottleneck: how slow collection actually is, where the throughput hides, and the three levers that move it. The benchmarks are first-hand from our collection floors; the dataset math comes from published sources.

Key Takeaways – DROID, one of the largest open manipulation datasets, is 76,000 episodes and about 350 hours of interaction collected across 13 institutions (arxiv.org/abs/2403.12945). An LLM ingests more tokens than that dataset’s total information content in seconds. – A trained operator produces 25 to 40 usable episodes per hour. A novice produces 8 to 12. The ramp takes two to three weeks. – QA rejects 10 to 30 percent of episodes. Most teams forget to budget for it. – The three scaling levers, in order of impact: operator retention, task and rig ergonomics, automated QA at ingest.

The Bottleneck, Quantified

The teleoperation bottleneck is the mismatch between how fast foundation models can consume demonstration data and how fast humans can produce it. Consider what “large” means in robotics. Open X-Embodiment pooled over one million trajectories from 22 embodiments, drawing on years of collection by dozens of labs (arxiv.org/abs/2310.08864). DROID took 76,000 episodes across 564 scenes and stands as a landmark effort. Meanwhile a single humanoid program we support wants 50,000 episodes per quarter, for one product.

Run the arithmetic. At 30 usable episodes per operator-hour, 50,000 episodes is roughly 1,700 operator-hours before rejection. Add a realistic 15 percent QA rejection and you are near 2,000 hours: one operator working full-time for a year, or ten operators for five weeks, assuming every one of them is already trained. They will not be.

Where the Throughput Actually Goes

Throughput loss in teleoperation programs comes from three places: the operator ramp curve, episode overhead, and QA rejection. Each one quietly halves your effective rate if ignored.

The operator ramp curve is the two-to-three-week climb from novice to trained throughput. Our benchmarks are consistent: new operators deliver 8 to 12 usable episodes per hour in weeks one and two, then 25 to 40 once bimanual coordination and reset discipline become automatic. That is a 3x difference, which makes operator churn the most expensive line item nobody tracks. Lose a trained operator and you do not lose a salary; you lose three weeks of peak yield.

Episode overhead is everything between demonstrations: scene resets, object randomization, rig checks, breaks. On contact-rich tasks with careful resets, overhead can consume 40 percent of a shift. Task card design and reset tooling (marked object regions, reset jigs) pull that down more than exhortation ever will.

QA rejection runs 20 to 30 percent on a new task or rig and 10 to 15 percent in a mature pipeline. Sync skew between the 50 Hz action log and 30 fps cameras is the top rejection reason we see, followed by hesitant trajectories and failed completions. Rejection discovered at training time, weeks after collection, is pure loss; rejection at ingest is a correctable signal.

The Three Levers That Move the Number

Scaling teleoperation data collection means attacking retention, ergonomics, and ingest QA, in that order.

Lever	Typical gain	Cost to implement
Operator retention and certification program	Sustains 25 to 40 eps/hr vs repeated novice resets	Low: pay, scheduling, progression path
Rig and task ergonomics (leader-follower feel, reset jigs)	20 to 40 percent throughput on contact-rich tasks	Medium: GELLO-class leaders ~$300/arm, jig fabrication
Automated QA at ingest (sync, smoothness, completion gates)	Recovers the 10 to 30 percent otherwise lost silently	Medium: one engineer-month, then near-zero marginal

A note on hardware, since teams often reach for it first. Moving from a $500 VR setup to a $20k ALOHA-class station improves fidelity on fine manipulation, and the original ACT results (80 to 90 percent success from about 50 demos, arxiv.org/abs/2304.13705) show what a well-matched rig enables. But hardware does not fix churn, and it does not fix silent QA loss. We have seen $300 GELLO leaders outproduce premium stations purely because the operators stayed.

A Worked Example: Sizing a 25,000-Episode Program

Program sizing is the exercise of converting an episode target into stations, headcount, and calendar time, with ramp and rejection built in rather than bolted on. Here is the arithmetic for a representative target of 25,000 QA-passed episodes in one quarter.

Start from the end. With a 15 percent rejection buffer, 25,000 passed episodes means collecting about 29,400. A trained operator averaging 30 usable episodes per hour, working six productive hours per shift, yields 180 episodes per day. Naively that is 164 operator-days, or three operators for the quarter. The naive answer is wrong in three ways.

First, ramp. If all three operators start as novices at 10 episodes per hour, the first three weeks produce a third of plan, which costs you roughly 5,000 episodes against the target. Either hire ahead of need or extend the calendar. Second, absence and churn. Floors run at 80 to 90 percent of rostered capacity over a quarter; one resignation among three operators is a 33 percent capacity event plus a new ramp curve. Third, task mix. Contact-rich bimanual tasks run at the bottom of the throughput range, and new tasks spike rejection to 20 to 30 percent for their first two weeks regardless of operator skill.

The sizing that actually holds: five operators across three stations (stations can run staggered shifts), a certification cohort starting two weeks before production, and a rejection buffer that starts at 25 percent for week one of each new task and decays to 15 percent. At $28 to $60 per all-in operator hour, the labor line for the quarter lands between roughly $27k and $70k depending on region and task complexity. That is the honest shape of the bottleneck: not impossible, just physical.

What This Means for Your Roadmap

The practical conclusion is that demonstration data needs the same planning rigor as compute. If your model roadmap says “100k episodes by Q2,” the collection plan behind it should specify stations, operator headcount with ramp assumptions, a rejection buffer of at least 15 percent, and an ingest QA gate that runs nightly. Teams that treat collection as an afterthought discover the bottleneck at the worst possible time: after the training cluster is reserved.

The full playbook, including rig selection tables and per-episode cost math, is in our complete guide to teleoperation data collection.

Next Step

Ready to scope your data program? Talk to our team.

Frequently Asked Questions

Why is teleoperation data the bottleneck in physical AI?

Because action-labeled interaction data cannot be scraped from the web; it must be produced by human operators at 25 to 40 usable episodes per hour at best. Model capacity and compute scale far faster than human demonstration throughput.

How fast can one operator collect demonstration data?

Our benchmarks: 8 to 12 usable episodes per hour for a novice, rising to 25 to 40 after two to three weeks of ramp. Contact-rich bimanual tasks sit at the low end, simple pick-place at the high end.

How much teleoperation data do VLA models need?

Multi-task training typically needs thousands of episodes per skill family. For reference, Open X-Embodiment pooled over one million trajectories from 22 embodiments, and single-task ACT policies can succeed with about 50 clean demonstrations.

What percentage of teleoperation data fails QA?

Expect 20 to 30 percent rejection on new tasks or rigs and 10 to 15 percent in a mature pipeline. The top causes are timestamp skew between streams, dropped frames, and failed task completions.

Can simulation replace teleoperation data?

Not yet for contact-rich manipulation. Simulation supplements real data and helps with evaluation, but sim-to-real gaps in contact dynamics and perception mean most production policies still train primarily on real teleoperated demonstrations.

Teleoperation Data Collection for Robot Learning: The Complete 2026 Guide

Posted on July 19, 2026July 21, 2026 by Sainath Gupta

Midway through a humanoid data program last year, our cost per usable episode rose almost 40 percent in a single week, and for two days nobody could explain it. Every rig passed its health checks. The QA gates were green. The answer had nothing to do with hardware or pipelines: two trained operators had left, their replacements were producing at novice rates, and the ramp math did the rest. Papers make demonstration collection look like a solved problem. In production, it is a staffing, hardware, and QA operation that most ML teams have never run before.

That week is this guide in miniature, and it carries the thesis the whole page argues: teleoperation data collection is an industrial operation, not an ML task, and it succeeds or fails on operational levers. It is part hardware engineering (leader arms, cameras, time sync), part human operations (hiring, training, and retaining operators), and part data engineering (formats, QA, versioning). Underestimate any one leg and the dataset comes out late, expensive, or unusable.

This guide covers the full stack: what teleoperation data collection actually is, the five main rig approaches and what they cost, the economics of operator labor, how demonstrations turn into trained policies, and the QA thresholds that separate usable datasets from expensive noise. The numbers on operator throughput, cost per hour, and rejection rates come from our own collection floors; the hardware and algorithm claims come from the primary literature (ALOHA, GELLO, UMI, DROID, Open-TeleVision), cited throughout.

TL;DR – Teleoperation data collection is the process of recording synchronized robot actions and sensor observations while a human remotely controls the robot through a task, producing demonstrations for imitation learning. – Five rig families dominate: leader-follower arms (ALOHA, ~$20k bimanual), low-cost kinematic leaders (GELLO, roughly $300 per arm), VR controllers/headsets (Quest 3, Vision Pro), 6-DoF desktop devices (SpaceMouse), and robot-free handheld capture (UMI). – Realistic all-in operator cost runs $28 to $60 per teleop hour. Novice operators produce 8 to 12 usable episodes per hour; trained operators reach 25 to 40 within two to three weeks. – Expect 10 to 30 percent QA rejection. Budget for it up front or your episode targets will slip. – 50 clean demonstrations trained with ACT reached 80 to 90 percent success on fine bimanual tasks in the original ALOHA work. Quality compounds; volume alone does not.

What Is Teleoperation Data Collection?

Teleoperation data collection is the process of recording a robot’s proprioceptive states, actions, and camera observations while a human operator remotely controls the robot through a task, producing time-aligned demonstration episodes used to train imitation learning policies. Each episode is a synchronized log: joint positions and velocities, commanded actions, gripper state, and multiple camera streams, typically stored in HDF5 or the LeRobot dataset format (github.com/huggingface/lerobot).

The entity chain matters for anyone mapping this space. Teleoperation produces demonstrations. Demonstrations feed imitation learning algorithms such as ACT (Action Chunking with Transformers, arxiv.org/abs/2304.13705) and Diffusion Policy (arxiv.org/abs/2303.04137). Those policies, scaled across tasks and embodiments, become the training substrate for vision-language-action (VLA) models. Open X-Embodiment aggregated over one million trajectories across 22 robot embodiments largely from teleoperated sources (arxiv.org/abs/2310.08864). No teleop data, no VLA.

A useful mental model: a teleop episode is worth exactly as much as its worst-synchronized stream. A 50 Hz action log paired with a camera that drifted 80 ms out of alignment teaches the policy that objects move before the hand does. That single failure mode wastes more training budget than any other we see.

Core Concepts and Data Modalities

A teleoperation dataset is defined by four things: the action space, the observation streams, the control and capture rates, and the episode structure. Get these right before collecting a single demo, because retrofitting a format after 10,000 episodes is miserable.

Action Space and Leader-Follower Mapping

Leader-follower joint mapping is the control scheme in which a human moves a small “leader” arm and a “follower” robot arm mirrors those joint positions in real time. ALOHA uses this directly: two WidowX-class leader arms drive two ViperX 300 followers, joint to joint, which gives operators immediate kinesthetic intuition and avoids inverse kinematics singularities (arxiv.org/abs/2304.13705). GELLO takes the same idea and rebuilds the leader from hobby-grade Dynamixel servos and printed parts for roughly $300 per arm (arxiv.org/abs/2309.13037). VR and SpaceMouse rigs instead command end-effector poses and rely on an IK solver, which is cheaper but introduces its own artifacts near joint limits.

Observation Streams: Mono, Stereo, Wrist, and Overhead

Observation streams are the camera and sensor feeds recorded alongside actions, and their placement determines what the policy can actually learn. The standard manipulation setup is one or two exocentric (scene) cameras plus one wrist camera per arm. Stereo pairs or depth add geometric grounding for tasks with occlusion. Our default rig records three RGB streams at 640×480, 30 fps, plus joint states at the control rate. Egocentric head cameras matter for humanoid teleop, where systems like Open-TeleVision stream an actively controlled head view into the operator’s headset (arxiv.org/abs/2407.01512).

Control Rate vs Camera Rate: The 50 Hz / 30 fps Problem

The rate mismatch problem is the timing gap created when actions are logged at 50 Hz but cameras capture at 30 fps, leaving no camera frame for two out of every five action steps. ACT-style pipelines handle this by training on the nearest or interpolated frame and predicting action chunks (sequences of 50 to 100 future actions) rather than single steps, which smooths over per-step jitter. What the algorithm cannot fix is unsynchronized clocks. Hardware-timestamp every stream at the source, log in one process or use a shared monotonic clock, and record the offsets. In our QA pipeline, any episode with inter-stream skew above 33 ms (one camera frame) is auto-flagged, and it is the single most common rejection reason on new rigs.

Episode Structure and Task Definition

An episode is one complete attempt at a defined task, from a randomized initial state to a labeled terminal state. Discipline here is what makes a dataset trainable: written task cards, explicit reset procedures, randomized object poses within a marked region, and a per-episode success label assigned at collection time. Datasets that skip initial-state randomization produce policies that memorize a table layout instead of learning a skill.

The Five Rig Approaches Compared

Rig choice is the decision that fixes your cost floor, your data quality ceiling, and which tasks you can collect at all. There is no universal winner; there is a right answer per task family and budget.

Approach	Example system	Hardware cost	Control fidelity	Operator ramp	Best for	Weakness
Bimanual leader-follower	ALOHA (Stanford)	~$20k; Mobile ALOHA ~$32k with base	Joint-level, 50 Hz, kinesthetic	Fast (days)	Fine bimanual manipulation	Cost per station; fixed embodiment
Low-cost kinematic leader	GELLO	~$300 per leader arm + robot	Joint-level, high	Fast	Scaling stations cheaply on arms you own	Build/maintenance effort; per-robot tuning
VR headset + controllers	Quest 3 rigs; Open-TeleVision style	$500 to $3,500 + robot	Pose-level via IK, medium-high	Medium (1 to 2 weeks)	Humanoids, mobile manipulation, immersive tasks	IK artifacts, some operator fatigue/sim sickness
6-DoF desktop device	3Dconnexion SpaceMouse	$150 to $450 + robot	Pose-level, low-medium	Slow for contact-rich work	Simple pick-place, lab prototyping	Slow, unintuitive for dexterous or bimanual tasks
Robot-free handheld	UMI gripper	Under $1,500 per device	Demonstrations without a robot	Fast	In-the-wild diversity, pre-robot data	Embodiment gap; SLAM-dependent pose quality

Three field notes that the spec sheets miss. First, DROID, one of the largest open manipulation datasets (76,000 episodes, 350 hours, 564 scenes), was collected with a Meta Quest 2 controller driving a Franka arm, proof that consumer VR hardware scales to serious datasets (arxiv.org/abs/2403.12945). Second, GELLO is the highest-throughput-per-dollar option if you already own the follower arms, but plan for a part-time technician; printed leaders drift and servos strip. Third, UMI (arxiv.org/abs/2402.10329) is the only approach that collects before your robot ships, which makes it a strong hedge for teams still finalizing hardware.

For a deeper head-to-head, see our companion piece on comparing teleoperation approaches.

Economics: What Teleoperation Data Actually Costs

The true cost of teleoperation data is operator hours multiplied by an all-in rate, divided by usable episode yield, and most budgets get all three numbers wrong. Hardware is the smaller line item; a $20k ALOHA station running two shifts amortizes to under $3 per collection hour over a year. Labor and QA dominate.

Our first-hand benchmarks, consistent across programs we run:

Cost and throughput factor	Typical range (our benchmarks)
All-in operator cost (wages, supervision, facility, hardware amortization)	$28 to $60 per teleop hour
Novice operator throughput (weeks 1 to 2)	8 to 12 usable episodes per hour
Trained operator throughput (week 3 onward)	25 to 40 usable episodes per hour
QA rejection rate (mature pipeline)	10 to 15 percent
QA rejection rate (new task or new rig, first 2 weeks)	20 to 30 percent
Effective cost per usable episode	roughly $0.90 to $6.00 depending on task complexity

Two implications fall out of that table. The operator ramp curve is a 3x throughput difference, so operator retention is a data-cost lever bigger than any hardware decision; churn resets you to novice yield. And QA rejection compounds against you: a 10,000-episode target with 25 percent rejection means collecting 13,300, not 10,000. We cover the bottleneck math in detail in why teleoperation data is the bottleneck in physical AI.

ROI framing for buyers: at $40 per hour all-in and 30 usable episodes per hour, a 50,000-episode dataset costs about $67k in collection labor before QA overhead. Compare that against three to six months of two ML engineers building and debugging an in-house collection operation, and outsourced collection frequently wins on time even when it ties on cost.

From Demonstrations to Policy: The Imitation Learning Pipeline

The imitation learning pipeline is the sequence that converts teleoperated demonstrations into an evaluated robot policy: collect episodes, convert to a training format, train ACT or a diffusion policy, then measure real-world success rate. The pipeline is short; the feedback loop is what teams underuse.

Collect and QA. Episodes stream into storage with per-episode metadata (task, operator, rig, success label). QA gates on sync skew, dropped frames, trajectory smoothness, and task success.
Convert. HDF5 or ROS 2 bags convert to LeRobot format for training and sharing on Hugging Face.
Train. ACT with action chunking (predicting ~100-step action sequences at 50 Hz) or Diffusion Policy for multimodal action distributions.
Evaluate. Fixed evaluation protocol: N rollouts per task with randomized initial states, report success rate.
Loop. Failure modes from evaluation define next week’s collection priorities.

The proof point worth anchoring on: the original ACT work reached 80 to 90 percent success on fine bimanual tasks such as cable routing and battery insertion from only 50 demonstrations per task (arxiv.org/abs/2304.13705). Demonstration quality and consistency, not raw volume, set that outcome. We have watched 400 sloppy episodes lose to 120 clean ones on the same task, same architecture. Collect less, reject more, retrain sooner.

Case Study Proof: Scaling a VLA Data Program

A humanoid foundation model team came to us needing 12,000 QA-passed bimanual episodes across 40 household tasks in ten weeks, after their in-house effort had produced 1,900 episodes in eight weeks with a 34 percent rejection rate. We stood up six leader-follower stations, ran a two-week operator certification program, and enforced automated sync and smoothness gates at ingest. By week four the floor was producing 1,400 usable episodes per week; the program closed at 12,600 QA-passed episodes with a 12 percent final rejection rate, and the team’s ACT baseline improved from 41 to 68 percent average task success on their internal eval. The full breakdown, including what failed in weeks one and two, is in the VLA case study.

Standing Up a Collection Program: The First 30 Days

A collection program launch is a 30-day sequence of hardware bring-up, protocol authoring, operator certification, and QA gate activation, in that order. Teams that compress or reorder it pay the cost back with interest in weeks five through eight. Our standard sequence:

Days 1 to 7: rigs and clocks. Assemble stations, calibrate cameras (record intrinsics and extrinsics per rig, per calibration), and prove time sync before anyone collects a demonstration. Log a physical sync event (a clapboard works) and verify every stream lands within one camera frame. A day spent here saves a re-collection later; skew discovered after 5,000 episodes is not fixable in post.

Days 8 to 14: protocol. Write task cards for the first five tasks: success criteria, reset procedure, randomization zones marked physically on the workspace, and expected episode duration. Pilot each card yourself before an operator touches it. Every ambiguity you leave in a task card becomes a labeling inconsistency in the dataset.

Days 15 to 28: certification. Run operators through reference tasks with a throughput bar (we use 20 usable episodes per hour) and a quality bar (QA pass rate above 80 percent) before production collection begins. Expect the standard ramp: 8 to 12 usable episodes per hour early, 25 to 40 after two to three weeks.

Day 30: gates on. Nightly automated QA becomes mandatory, with per-operator, per-rig reports every morning. From this point, the program self-corrects.

Build vs Buy: The Honest Decision Criteria

The build-versus-buy question comes down to whether demonstration collection is a competency your team needs to own or a supply problem it needs solved. Building makes sense in three cases: your embodiment is unusual enough that no vendor has matching rigs, your data requirements are small (under roughly 5,000 episodes), or collection know-how is itself part of your product thesis. Buying makes sense when the deadline is the constraint. The math from the economics section applies either way, but in-house programs carry two costs that rarely appear in the comparison spreadsheet: the two to four engineer-months of pipeline and QA tooling before the first reliable episode, and the management overhead of running an operations floor inside an ML organization. We are a vendor, so discount our view accordingly; then ask any team that has run both models which cost they underestimated.

Free Resource: Teleoperation Data RFP Scorecard

A vendor RFP scorecard is a weighted rubric for comparing teleoperation data providers on the criteria that actually predict dataset quality. Ours scores ten dimensions, including sync guarantees, QA rejection transparency, operator training programs, format deliverables, and per-episode pricing disclosure. Most vendors will not publish cost per hour; make them fill in the sheet. [Download the RFP scorecard (XLSX)] and use it on us too.

Next Step

If you are scoping a demonstration data program, two options. Download 10 free teleoperation episodes (HDF5 plus synchronized MP4, three camera views, 50 Hz actions) and run them through your training stack. Or book a 30-minute scoping call with our data operations team and leave with a per-episode cost estimate for your task list. [Download Sample Data] | [Book a Demo]

Frequently Asked Questions

What is teleoperation data collection?

Teleoperation data collection is the recording of synchronized robot actions, joint states, and camera observations while a human operator remotely controls the robot through a task. The resulting episodes train imitation learning policies such as ACT and diffusion policies, which underpin VLA models.

How much does teleoperation data collection cost?

Our all-in benchmark is $28 to $60 per operator hour, covering wages, supervision, facility, and hardware amortization. With trained operators producing 25 to 40 usable episodes per hour, effective cost lands between roughly $0.90 and $6.00 per usable episode depending on task complexity.

How many demonstrations do I need to train a policy?

Fewer than most teams assume, if quality is high. The original ALOHA/ACT results reached 80 to 90 percent success on fine bimanual tasks with about 50 demonstrations per task. Multi-task and VLA training raises the requirement into the thousands per skill family, which is where collection economics start to dominate.

What is the best rig for teleoperation data collection?

It depends on the task family and budget, per the comparison table above. Fine bimanual manipulation favors leader-follower rigs (ALOHA at ~$20k, or GELLO leaders at ~$300 per arm on robots you own). Humanoid and mobile tasks favor VR teleop. Robot-free UMI capture wins when hardware has not shipped yet.

Why do teleoperation datasets get rejected in QA?

The main causes are inter-stream time skew above one camera frame, dropped frames, jerky or hesitant trajectories, failed task completions, and reset protocol violations. Expect 20 to 30 percent rejection in the first two weeks on a new task and 10 to 15 percent once the pipeline matures.

Can I collect demonstration data without a robot?

Yes. UMI-style handheld grippers record demonstrations with a wrist-mounted camera and SLAM-derived poses, no robot required. The trade-off is an embodiment gap when transferring to your platform, so most teams use it to supplement, not replace, on-robot teleop.

What format should teleoperation data be stored in?

HDF5 per episode and the LeRobot dataset format are the current de facto standards; ROS 2 bags are common at capture time. Whatever the container, require hardware timestamps on every stream and per-episode metadata (task, operator ID, rig ID, success label).

Comparing Egocentric Data Collection for Robotics Approaches: Pros, Cons and Costs

Posted on July 14, 2026July 16, 2026 by Sainath Gupta

There is no best egocentric capture rig, and shopping for one is how robotics teams burn their first data budget. I have built or debugged every rig family in this post, and the pattern behind the expensive mistakes I get called in to fix is always the same: a team copied another lab’s hardware without copying the training objective that made the hardware correct. That is the thesis this comparison defends: rig choice is a downstream consequence of your training mechanism, and any comparison that ranks rigs without naming the mechanism is selling you something.

The confusion is understandable. The hardware landscape moved fast: research glasses like Aria Gen 2 appeared alongside consumer devices like the Meta Quest 3 that happen to make decent capture rigs, while GoPros and RealSense cameras have been quietly strapped to helmets for years. Each option makes a different trade between image quality, sensor completeness, cost, and how much post-processing pain you inherit.

This post compares the four egocentric rig families we run in production, plus teleoperation and simulation as the two alternatives buyers always ask about. You get pros, cons, our per-hour cost benchmarks, and a decision matrix that maps rigs to training objectives.

At DexSet we operate all of these rigs daily across egocentric, exocentric, and teleoperation programs, so the failure modes below are ones we have paid for personally.

Key Takeaways – Rig choice should follow training mechanism: mono action cams for encoder pretraining, instrumented glasses for action retargeting, stereo depth helmets for metric 3D, teleop for embodiment-matched fine-tuning. – Our finished-hour cost benchmarks: GoPro mono $15 to $20, Quest 3 $18 to $26, RealSense helmet $20 to $30, Aria-class glasses $25 to $40, teleoperation $28 to $60. – Cheap capture is not cheap data. Uncalibrated mono footage shifts cost from hardware to annotation and 3D lifting compute. – Simulation complements rather than replaces real capture; the sim-to-real gap in contact-rich manipulation remains the limiting factor. – Pilot 50 to 100 hours on your shortlisted rig and train on it before committing to a full program.

What Counts as an Egocentric Collection Approach?

An egocentric collection approach is the combination of a wearable sensor rig, a capture protocol, and a post-processing pipeline that together produce first-person training episodes. The rig gets the attention, but all three layers set your cost. A $500 headset with a sloppy protocol produces more expensive data than a disciplined GoPro program, once you price the rejected hours.

For orientation across the whole field of modalities, hardware, and economics, see our pillar: The Complete Guide to Egocentric Data Collection for Robotics. This post narrows to the comparison question.

The Master Comparison Table

Approach	Hardware Cost	Sensors You Get	Finished Cost/Hr (Our Benchmarks)	Biggest Strength	Biggest Weakness
GoPro head/chest mount	$350 to $550	Mono RGB (wide FOV), IMU	$15 to $20	Cheapest scale; rugged	No depth, no hand pose; annotation-heavy
Meta Quest 3	~$500	Stereo passthrough, IMU, hand tracking	$18 to $26	Built-in 3D hand pose	Constrained capture access; middling image quality
RealSense D435i/D455 helmet	$700 to $1,200 built	Stereo IR + RGB, active depth, IMU	$20 to $30	Metric depth for manipulation	Custom build; per-rig calibration burden
Aria-class research glasses	Research program access	Multi-camera RGB + SLAM, 2 IMUs, eye tracking, hand tracking	$25 to $40	Factory calibration, gaze, richest streams	Fleet scaling gated by program access
Teleoperation (ALOHA-class)	~$20K+ per station	Robot proprioception + cameras, exact actions	$28 to $60	Perfect embodiment match	Throughput ceiling; capex and lab ops
Simulation	Compute cost	Anything you render	$1 to $5 equivalent	Infinite variation, free labels	Sim-to-real gap in contacts and materials

Costs are DexSet production benchmarks for QA-passed hours at typical annotation depths; your protocol and rejection rate will move them.

GoPro Rigs: The Volume Play

A GoPro-based rig is a mono wide-angle action camera on a head or chest mount, and it remains the highest-throughput, lowest-cost way to collect egocentric video. We run 4K/60 or 2.7K/120 depending on task speed. The cameras survive kitchens, warehouses, and weather, batteries swap in seconds, and collectors need five minutes of training.

The costs arrive downstream. Mono RGB has no metric depth, so any 3D understanding must be lifted with structure-from-motion or learned depth, and hand pose must be estimated rather than measured. For encoder pretraining on thousands of hours, that trade is usually correct; the Ego4D corpus proved how far lightly instrumented first-person video can go (arXiv:2110.07058). For action retargeting, it is usually wrong.

Choose when: you need thousands of hours for representation learning and can tolerate estimated 3D. Avoid when: your pipeline consumes metric hand trajectories.

Quest 3: The Accidental Capture Device

The Meta Quest 3 is a ~$500 consumer VR headset whose passthrough cameras and native hand tracking make it a surprisingly capable egocentric rig. The hand tracking is the point: you get approximate 3D hand pose at capture time, free, on hardware a college student can buy retail. Teams also reuse the same headset as a teleoperation interface, which keeps the hardware pool simple.

The weaknesses are real. Access to raw passthrough streams is constrained by platform policy, effective image quality trails dedicated cameras, and wearing a headset for long capture shifts changes how people move; collectors are measurably more hesitant in headset than in glasses, and that hesitancy shows up in the motion statistics of your dataset.

Choose when: hand pose matters, budget is tight, and sessions are short. Avoid when: you need long natural sessions or top-tier image quality.

RealSense Helmet Rigs: Metric Depth on a Budget

A RealSense helmet rig mounts an Intel D435i or D455 stereo depth camera on headgear, giving hardware-synced stereo, active depth, and an onboard IMU for roughly $700 to $1,200 built. The D455’s 95 mm baseline improves depth accuracy at room distances over the D435i’s 50 mm; we pick per task family. When your consumers need metric 3D (grasp point estimation, sim scene reconstruction, depth-conditioned policies), this is the price-performance sweet spot.

You pay in operations. These are custom builds, so calibration is on you, mounts loosen, IR projectors misbehave in sunlight, and USB bandwidth is a daily negotiation. Budget real engineering time or the depth streams silently degrade.

Choose when: metric depth is a hard requirement. Avoid when: you cannot staff rig maintenance.

Aria-Class Glasses: The Quality Ceiling

Aria Gen 2 research glasses are purpose-built egocentric capture devices with calibrated multi-camera arrays, dual IMUs, eye tracking, and on-device machine perception (projectaria.com). Data arrives factory-calibrated with services for SLAM and hand tracking, which deletes whole stages of your post-processing pipeline. EgoExo4D was collected on Aria hardware (arXiv:2311.18259), and retargeting work like EgoMimic used Aria capture as its human-data source (arXiv:2410.24221). Collectors also behave naturally in glasses, which matters more than teams expect.

The constraint is access: these are research program devices rather than retail products, so fleet scaling depends on program terms rather than a purchase order.

Choose when: you need retargeting-grade data with gaze and hand pose at research quality. Avoid when: your plan requires buying 200 units next month.

Teleop and Sim: The Two Non-Egocentric Alternatives

Teleoperation and simulation are the alternatives every budget conversation reaches, and both are complements rather than substitutes. Teleoperation through ALOHA-class rigs (arXiv:2304.13705) is the only approach that produces exact robot-embodiment actions, which is why fine-tuning sets are teleop even when pretraining is egocentric; the LeRobot ecosystem has standardized much of this tooling (github.com/huggingface/lerobot). Simulation generates unlimited labeled variation at $1 to $5 per equivalent hour, and remains weakest exactly where manipulation is hardest: contact dynamics, deformables, and material appearance.

Decision Matrix: Match the Rig to the Mechanism

Your Training Objective	First Choice	Second Choice
Visual encoder pretraining at scale	GoPro mono	Quest 3
Action retargeting to grippers	Aria-class glasses	Quest 3
Depth-conditioned manipulation	RealSense helmet	Aria-class glasses
Embodiment-matched fine-tuning	Teleoperation	(no substitute)
Domain randomization / rare events	Simulation	Sim + real mix

Choosing From Here

The full economics, QA gates, and rig field notes behind this comparison are in the pillar guide linked above. If you would rather see the data than read about it, book a demo and we will put sample episodes from three different rigs side by side on a call.

Frequently Asked Questions

Which egocentric capture rig is cheapest per hour?

GoPro-based mono rigs are cheapest in our benchmarks at $15 to $20 per finished hour, but they shift cost into annotation and 3D lifting because they capture no depth or hand pose.

Is the Meta Quest 3 good for robotics data collection?

Yes, for short sessions where 3D hand pose matters and budgets are tight. Its native hand tracking provides approximate pose at capture time, though passthrough access limits and image quality keep it below dedicated rigs for long natural capture.

When do I need stereo depth instead of mono video?

When downstream consumers need metric 3D: grasp point estimation, scene reconstruction for sim, or depth-conditioned policies. RealSense D435i/D455 helmet rigs are the common budget path; Aria-class glasses the research-grade one.

Can simulation replace egocentric data collection?

No. Simulation adds cheap variation and labels but still misses contact dynamics and material realism in manipulation, so real egocentric and teleop data remain necessary anchors.

How should I choose between these approaches?

Derive the choice from your training mechanism: mono for encoder pretraining, instrumented glasses or Quest 3 for retargeting, stereo helmets for metric depth, teleoperation for embodiment-matched fine-tuning. Pilot 50 to 100 hours before scaling.

Why Egocentric Data Collection for Robotics Is the Biggest Bottleneck in Physical AI

Posted on July 13, 2026July 16, 2026 by Enoch Pakanati

In the kitchen bay of our capture floor, a collector claps once in front of her helmet camera, waits for the sync check to pass on the wall monitor, and begins unloading a dishwasher using only grasps a parallel-jaw gripper could copy. By the end of her shift she will have produced more usable first-person manipulation data than one of our bimanual teleoperation stations yields in a week. Nothing about that scene is exotic. The exotic thing is how rare it still is.

It is rare for a structural reason. Text and images were exhaust from human activity; someone else already paid to produce them. First-person manipulation data is not exhaust. Nobody films their hands doing the dishes with calibrated stereo cameras and synchronized IMUs, so every hour has to be manufactured deliberately, by someone wearing a rig or driving a teleoperation station, and then inspected before it is fit to train on. The largest open robot manipulation dataset, Open X-Embodiment, took 21 institutions pooling 22 embodiments to reach roughly a million trajectories (arXiv:2310.08864), while language models trained on trillions of tokens.

This post argues one thesis: the binding constraint on physical AI right now is the supply of QA-passed egocentric hours, not compute and not architecture. You will get the arithmetic behind that claim, our cost benchmarks for the three ways teams try to break the constraint, and a checklist for scaling collection without torching your budget.

We spend our days at DexSet manufacturing exactly this data: egocentric, exocentric, and teleoperation streams for VLA and humanoid teams. The numbers below come from our production logs, not analyst reports.

Key Takeaways

Robot learning is data-bound: internet-scale text corpora dwarf the largest robot manipulation datasets by many orders of magnitude, and almost no internet video is first-person with recoverable actions.
Teleoperation alone cannot close the gap. A well-run bimanual station yields roughly 15 to 25 usable episode-hours per week; at $28 to $60 per hour, 10,000 hours is a multi-year, seven-figure program.
Egocentric human capture shifts the curve: $15 to $40 per hour in our benchmarks, 3 to 5x the collection rate per operator, and it scales with people instead of robot fleets.
The constraint is not raw hours, it is QA-passed hours. Plan for 15 to 25 percent of captured footage to fail quality gates.
Scaling works when you treat collection as a manufacturing process: fixed protocols, instrumented rigs, sampled QA, and annotation depth matched to the training objective.

Why Data, Not Compute, Is the Binding Constraint

The binding constraint in a training pipeline is the input you cannot buy more of at market price, and for physical AI that input is embodiment-relevant data. Compute is rentable by the hour. Architectures are published within months of working. But there is no marketplace where you can order 10,000 hours of first-person manipulation with hand pose annotations for next quarter, because until recently the industry that manufactures it did not exist.

Look at what the flagship models actually consumed. RT-2 leaned on internet-scale vision-language pretraining precisely because robot data was scarce (arXiv:2307.15818). OpenVLA trained on 970K robot episodes from Open X-Embodiment (arXiv:2406.09246). π₀ combined internet-scale pretraining with a proprietary dexterous manipulation corpus (arXiv:2410.24164). The pattern across all three: model capability tracked data supply, and every team supplemented public data with expensive private collection.

Meanwhile the largest first-person video corpus in existence, Ego4D, holds 3,670 hours (arXiv:2110.07058). Impressive for a research effort, and still less footage than a single mid-size YouTube channel uploads in a year. For robotics, most of it lacks the calibration and action annotations that policy learning needs. The shelf is nearly empty. Whoever wants the data has to make it.

The Teleoperation Ceiling

The teleoperation ceiling is the practical throughput limit of collecting robot demonstrations through human-controlled rigs, and it is lower than most roadmaps assume. ALOHA-style bimanual stations (arXiv:2304.13705) made teleop dramatically cheaper to build; the ALOHA rig famously targeted a ~$20K bill of materials against six-figure predecessors. What they did not change is the physics of throughput.

In our teleop operations, a trained operator on a healthy rig produces 15 to 25 usable episode-hours per week once you subtract resets, calibration, failed episodes, and operator fatigue. Push operators harder and quality drops before throughput rises; fine manipulation under fatigue produces exactly the jerky, hesitant trajectories you do not want a policy imitating.

Now run the arithmetic for a 10,000-hour corpus, which is a modest target by foundation model standards:

Approach	Cost per QA-passed hour (our benchmarks)	Realistic throughput	Time to 10,000 hours	Program cost
Teleoperation, 5 bimanual rigs	$28 to $60	75 to 125 hrs/week	1.5 to 2.5 years	$280K to $600K
Teleoperation, 20 rigs	$28 to $60	300 to 500 hrs/week	5 to 8 months	$280K to $600K + fleet capex
Egocentric human capture, 20 collectors	$15 to $40	400 to 700 hrs/week	4 to 6 months	$150K to $400K

The 20-rig teleop row hides the real problem: rig fleets need robot hardware, lab space, maintenance engineers, and operator pipelines. Twenty humans wearing $500 to $1,200 capture rigs need a protocol document and a QA queue. That asymmetry is the entire argument for egocentric capture as the scaling layer, with teleop reserved for the embodiment-matched fine-tuning set. Systems like EgoMimic have shown the combined recipe outperforms robot data alone (arXiv:2410.24221).

Why Egocentric Capture Is Hard to Scale Anyway

Scaling egocentric collection means turning a research-style capture activity into a manufacturing process with yield metrics, and most attempts stall on the same four problems.

Yield, not volume. Raw hours are easy; QA-passed hours are the product. Across our programs, 15 to 25 percent of captured footage fails gates for motion blur, dropped IMU packets, sync drift, or off-protocol behavior. Teams that do not measure yield discover it at training time, which is the most expensive place to discover it.

Calibration debt. Every uncalibrated GoPro hour you collect today is an hour of 3D lifting problems you inherit later. Devices like Aria carry factory calibration and machine perception services (projectaria.com); consumer rigs need per-unit calibration routines and periodic re-checks, and skipping them is the most common false economy we see.

Annotation as the cost center. Capture at $15 to $22 per hour feels cheap until hand pose and object tracks push finished cost to $30 to $40. The fix is not cheaper annotation; it is refusing to annotate hours your training plan does not need at that depth.

Protocol drift. Thirty collectors interpret “tidy the workspace” thirty ways. Without scripted task cards, fixed start states, and outcome labels, you end up with footage instead of episodes. The LeRobot dataset conventions (github.com/huggingface/lerobot) are a useful forcing function here: if a capture cannot be expressed as episodes with consistent schema, it probably was not collected with enough discipline.

A Scaling Checklist That Survives Contact With Reality

A scaling checklist is only useful if each item maps to a failure we have actually watched happen, so here is ours:

Define the training mechanism first (encoder pretraining, retargeting, co-training), then derive modalities and annotation depth from it.
Set a QA yield target (we hold vendors, including ourselves, to 80 percent+ first-pass) and sample-inspect every batch.
Match camera height and field of view to the deployment embodiment before collecting hour one.
Script tasks as episodes: start state, goal, outcome label, retry policy.
Calibrate on a schedule, not on failure.
Track cost per QA-passed hour weekly; it is the only number that predicts your budget.
Pilot 50 to 100 hours and train on them before committing to thousands.

If you want the full treatment of rigs, modalities, and economics behind this checklist, it lives in our pillar guide: The Complete Guide to Egocentric Data Collection for Robotics.

Frequently Asked Questions

Why is data the bottleneck in physical AI rather than compute?

Compute and architectures are purchasable and published, while first-person manipulation data with recoverable actions barely exists on the internet and must be manufactured deliberately. The largest robot datasets hold on the order of a million trajectories, orders of magnitude below the corpora that trained large language models.

How fast can teleoperation generate training data?

In DexSet’s operations, a trained operator on a bimanual rig produces roughly 15 to 25 usable episode-hours per week after resets, failures, and fatigue. Scaling further requires more robot rigs, lab space, and maintenance, which is why teleop fleets scale slowly.

How much cheaper is egocentric capture than teleoperation?

Our benchmarks put QA-passed egocentric capture at $15 to $40 per hour depending on annotation depth, against $28 to $60 per hour for teleoperation, with 3 to 5 times higher collection throughput per operator.

Does egocentric human data remove the need for robot data?

No. The embodiment gap between human hands and robot grippers means policies still need robot-embodiment demonstrations. The strongest published results co-train on both, as in EgoMimic.

What QA failure rate should I budget for?

Plan for 15 to 25 percent of raw captured footage to fail quality gates for blur, sync drift, sensor dropouts, or protocol violations.

From Bottleneck to Plan

If your roadmap has a data line item and a question mark next to it, start with the pillar guide above, then request a sample batch from us. We ship real episodes with real QA reports, and the pricing table is on the first page, not behind a discovery call.