Skip to main content

Dexset

dexset
Menu
Request a Demo

Tag: Teleoperation

Why Teleoperation Data Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

In a budget review we sat in on last quarter, a founder approved a seven-figure compute reservation in about ten minutes, then spent the next hour interrogating a $70k line for demonstration data. The compute felt legible: dollars per GPU-hour, a market price. The data line had no equivalent, because nobody in the room could say what an hour of human demonstration work actually yields.

That asymmetry is the subject of this post, and the thesis is blunt: physical AI is not compute-bound, it is demo-bound. Language models trained on the internet. Vision models trained on the internet. Robot foundation models have no internet of action-labeled data, so every demonstration a VLA model consumes had to be manufactured: a person, at a rig, doing a task, one episode at a time. That production rate, not FLOPs, sets the pace of the field.

The gap exists because physical interaction data cannot be scraped. YouTube shows what hands do but not the joint torques, gripper commands, and proprioceptive states a policy needs as action labels. Simulation helps but leaks reality at every contact. That leaves teleoperation as the primary source of ground-truth action data, and teleoperation runs at human speed.

This post puts numbers on the bottleneck: how slow collection actually is, where the throughput hides, and the three levers that move it. The benchmarks are first-hand from our collection floors; the dataset math comes from published sources.

Key Takeaways – DROID, one of the largest open manipulation datasets, is 76,000 episodes and about 350 hours of interaction collected across 13 institutions (arxiv.org/abs/2403.12945). An LLM ingests more tokens than that dataset’s total information content in seconds. – A trained operator produces 25 to 40 usable episodes per hour. A novice produces 8 to 12. The ramp takes two to three weeks. – QA rejects 10 to 30 percent of episodes. Most teams forget to budget for it. – The three scaling levers, in order of impact: operator retention, task and rig ergonomics, automated QA at ingest.

The Bottleneck, Quantified

The teleoperation bottleneck is the mismatch between how fast foundation models can consume demonstration data and how fast humans can produce it. Consider what “large” means in robotics. Open X-Embodiment pooled over one million trajectories from 22 embodiments, drawing on years of collection by dozens of labs (arxiv.org/abs/2310.08864). DROID took 76,000 episodes across 564 scenes and stands as a landmark effort. Meanwhile a single humanoid program we support wants 50,000 episodes per quarter, for one product.

Run the arithmetic. At 30 usable episodes per operator-hour, 50,000 episodes is roughly 1,700 operator-hours before rejection. Add a realistic 15 percent QA rejection and you are near 2,000 hours: one operator working full-time for a year, or ten operators for five weeks, assuming every one of them is already trained. They will not be.

Where the Throughput Actually Goes

Throughput loss in teleoperation programs comes from three places: the operator ramp curve, episode overhead, and QA rejection. Each one quietly halves your effective rate if ignored.

The operator ramp curve is the two-to-three-week climb from novice to trained throughput. Our benchmarks are consistent: new operators deliver 8 to 12 usable episodes per hour in weeks one and two, then 25 to 40 once bimanual coordination and reset discipline become automatic. That is a 3x difference, which makes operator churn the most expensive line item nobody tracks. Lose a trained operator and you do not lose a salary; you lose three weeks of peak yield.

Episode overhead is everything between demonstrations: scene resets, object randomization, rig checks, breaks. On contact-rich tasks with careful resets, overhead can consume 40 percent of a shift. Task card design and reset tooling (marked object regions, reset jigs) pull that down more than exhortation ever will.

QA rejection runs 20 to 30 percent on a new task or rig and 10 to 15 percent in a mature pipeline. Sync skew between the 50 Hz action log and 30 fps cameras is the top rejection reason we see, followed by hesitant trajectories and failed completions. Rejection discovered at training time, weeks after collection, is pure loss; rejection at ingest is a correctable signal.

The Three Levers That Move the Number

Scaling teleoperation data collection means attacking retention, ergonomics, and ingest QA, in that order.

Lever Typical gain Cost to implement
Operator retention and certification program Sustains 25 to 40 eps/hr vs repeated novice resets Low: pay, scheduling, progression path
Rig and task ergonomics (leader-follower feel, reset jigs) 20 to 40 percent throughput on contact-rich tasks Medium: GELLO-class leaders ~$300/arm, jig fabrication
Automated QA at ingest (sync, smoothness, completion gates) Recovers the 10 to 30 percent otherwise lost silently Medium: one engineer-month, then near-zero marginal

A note on hardware, since teams often reach for it first. Moving from a $500 VR setup to a $20k ALOHA-class station improves fidelity on fine manipulation, and the original ACT results (80 to 90 percent success from about 50 demos, arxiv.org/abs/2304.13705) show what a well-matched rig enables. But hardware does not fix churn, and it does not fix silent QA loss. We have seen $300 GELLO leaders outproduce premium stations purely because the operators stayed.

A Worked Example: Sizing a 25,000-Episode Program

Program sizing is the exercise of converting an episode target into stations, headcount, and calendar time, with ramp and rejection built in rather than bolted on. Here is the arithmetic for a representative target of 25,000 QA-passed episodes in one quarter.

Start from the end. With a 15 percent rejection buffer, 25,000 passed episodes means collecting about 29,400. A trained operator averaging 30 usable episodes per hour, working six productive hours per shift, yields 180 episodes per day. Naively that is 164 operator-days, or three operators for the quarter. The naive answer is wrong in three ways.

First, ramp. If all three operators start as novices at 10 episodes per hour, the first three weeks produce a third of plan, which costs you roughly 5,000 episodes against the target. Either hire ahead of need or extend the calendar. Second, absence and churn. Floors run at 80 to 90 percent of rostered capacity over a quarter; one resignation among three operators is a 33 percent capacity event plus a new ramp curve. Third, task mix. Contact-rich bimanual tasks run at the bottom of the throughput range, and new tasks spike rejection to 20 to 30 percent for their first two weeks regardless of operator skill.

The sizing that actually holds: five operators across three stations (stations can run staggered shifts), a certification cohort starting two weeks before production, and a rejection buffer that starts at 25 percent for week one of each new task and decays to 15 percent. At $28 to $60 per all-in operator hour, the labor line for the quarter lands between roughly $27k and $70k depending on region and task complexity. That is the honest shape of the bottleneck: not impossible, just physical.

What This Means for Your Roadmap

The practical conclusion is that demonstration data needs the same planning rigor as compute. If your model roadmap says “100k episodes by Q2,” the collection plan behind it should specify stations, operator headcount with ramp assumptions, a rejection buffer of at least 15 percent, and an ingest QA gate that runs nightly. Teams that treat collection as an afterthought discover the bottleneck at the worst possible time: after the training cluster is reserved.

The full playbook, including rig selection tables and per-episode cost math, is in our complete guide to teleoperation data collection.

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

Why is teleoperation data the bottleneck in physical AI?

Because action-labeled interaction data cannot be scraped from the web; it must be produced by human operators at 25 to 40 usable episodes per hour at best. Model capacity and compute scale far faster than human demonstration throughput.

Our benchmarks: 8 to 12 usable episodes per hour for a novice, rising to 25 to 40 after two to three weeks of ramp. Contact-rich bimanual tasks sit at the low end, simple pick-place at the high end.

Multi-task training typically needs thousands of episodes per skill family. For reference, Open X-Embodiment pooled over one million trajectories from 22 embodiments, and single-task ACT policies can succeed with about 50 clean demonstrations.

Expect 20 to 30 percent rejection on new tasks or rigs and 10 to 15 percent in a mature pipeline. The top causes are timestamp skew between streams, dropped frames, and failed task completions.

Not yet for contact-rich manipulation. Simulation supplements real data and helps with evaluation, but sim-to-real gaps in contact dynamics and perception mean most production policies still train primarily on real teleoperated demonstrations.

Teleoperation Data Collection for Robot Learning: The Complete 2026 Guide

Posted on by Sainath Gupta

Midway through a humanoid data program last year, our cost per usable episode rose almost 40 percent in a single week, and for two days nobody could explain it. Every rig passed its health checks. The QA gates were green. The answer had nothing to do with hardware or pipelines: two trained operators had left, their replacements were producing at novice rates, and the ramp math did the rest. Papers make demonstration collection look like a solved problem. In production, it is a staffing, hardware, and QA operation that most ML teams have never run before.

That week is this guide in miniature, and it carries the thesis the whole page argues: teleoperation data collection is an industrial operation, not an ML task, and it succeeds or fails on operational levers. It is part hardware engineering (leader arms, cameras, time sync), part human operations (hiring, training, and retaining operators), and part data engineering (formats, QA, versioning). Underestimate any one leg and the dataset comes out late, expensive, or unusable.

This guide covers the full stack: what teleoperation data collection actually is, the five main rig approaches and what they cost, the economics of operator labor, how demonstrations turn into trained policies, and the QA thresholds that separate usable datasets from expensive noise. The numbers on operator throughput, cost per hour, and rejection rates come from our own collection floors; the hardware and algorithm claims come from the primary literature (ALOHA, GELLO, UMI, DROID, Open-TeleVision), cited throughout.

TL;DR – Teleoperation data collection is the process of recording synchronized robot actions and sensor observations while a human remotely controls the robot through a task, producing demonstrations for imitation learning. – Five rig families dominate: leader-follower arms (ALOHA, ~$20k bimanual), low-cost kinematic leaders (GELLO, roughly $300 per arm), VR controllers/headsets (Quest 3, Vision Pro), 6-DoF desktop devices (SpaceMouse), and robot-free handheld capture (UMI). – Realistic all-in operator cost runs $28 to $60 per teleop hour. Novice operators produce 8 to 12 usable episodes per hour; trained operators reach 25 to 40 within two to three weeks. – Expect 10 to 30 percent QA rejection. Budget for it up front or your episode targets will slip. – 50 clean demonstrations trained with ACT reached 80 to 90 percent success on fine bimanual tasks in the original ALOHA work. Quality compounds; volume alone does not.

What Is Teleoperation Data Collection?

Teleoperation data collection is the process of recording a robot’s proprioceptive states, actions, and camera observations while a human operator remotely controls the robot through a task, producing time-aligned demonstration episodes used to train imitation learning policies. Each episode is a synchronized log: joint positions and velocities, commanded actions, gripper state, and multiple camera streams, typically stored in HDF5 or the LeRobot dataset format (github.com/huggingface/lerobot).

The entity chain matters for anyone mapping this space. Teleoperation produces demonstrations. Demonstrations feed imitation learning algorithms such as ACT (Action Chunking with Transformers, arxiv.org/abs/2304.13705) and Diffusion Policy (arxiv.org/abs/2303.04137). Those policies, scaled across tasks and embodiments, become the training substrate for vision-language-action (VLA) models. Open X-Embodiment aggregated over one million trajectories across 22 robot embodiments largely from teleoperated sources (arxiv.org/abs/2310.08864). No teleop data, no VLA.

A useful mental model: a teleop episode is worth exactly as much as its worst-synchronized stream. A 50 Hz action log paired with a camera that drifted 80 ms out of alignment teaches the policy that objects move before the hand does. That single failure mode wastes more training budget than any other we see.

Core Concepts and Data Modalities

A teleoperation dataset is defined by four things: the action space, the observation streams, the control and capture rates, and the episode structure. Get these right before collecting a single demo, because retrofitting a format after 10,000 episodes is miserable.

Action Space and Leader-Follower Mapping

Leader-follower joint mapping is the control scheme in which a human moves a small “leader” arm and a “follower” robot arm mirrors those joint positions in real time. ALOHA uses this directly: two WidowX-class leader arms drive two ViperX 300 followers, joint to joint, which gives operators immediate kinesthetic intuition and avoids inverse kinematics singularities (arxiv.org/abs/2304.13705). GELLO takes the same idea and rebuilds the leader from hobby-grade Dynamixel servos and printed parts for roughly $300 per arm (arxiv.org/abs/2309.13037). VR and SpaceMouse rigs instead command end-effector poses and rely on an IK solver, which is cheaper but introduces its own artifacts near joint limits.

Observation Streams: Mono, Stereo, Wrist, and Overhead

Observation streams are the camera and sensor feeds recorded alongside actions, and their placement determines what the policy can actually learn. The standard manipulation setup is one or two exocentric (scene) cameras plus one wrist camera per arm. Stereo pairs or depth add geometric grounding for tasks with occlusion. Our default rig records three RGB streams at 640×480, 30 fps, plus joint states at the control rate. Egocentric head cameras matter for humanoid teleop, where systems like Open-TeleVision stream an actively controlled head view into the operator’s headset (arxiv.org/abs/2407.01512).

Control Rate vs Camera Rate: The 50 Hz / 30 fps Problem

The rate mismatch problem is the timing gap created when actions are logged at 50 Hz but cameras capture at 30 fps, leaving no camera frame for two out of every five action steps. ACT-style pipelines handle this by training on the nearest or interpolated frame and predicting action chunks (sequences of 50 to 100 future actions) rather than single steps, which smooths over per-step jitter. What the algorithm cannot fix is unsynchronized clocks. Hardware-timestamp every stream at the source, log in one process or use a shared monotonic clock, and record the offsets. In our QA pipeline, any episode with inter-stream skew above 33 ms (one camera frame) is auto-flagged, and it is the single most common rejection reason on new rigs.

Episode Structure and Task Definition

An episode is one complete attempt at a defined task, from a randomized initial state to a labeled terminal state. Discipline here is what makes a dataset trainable: written task cards, explicit reset procedures, randomized object poses within a marked region, and a per-episode success label assigned at collection time. Datasets that skip initial-state randomization produce policies that memorize a table layout instead of learning a skill.

The Five Rig Approaches Compared

Rig choice is the decision that fixes your cost floor, your data quality ceiling, and which tasks you can collect at all. There is no universal winner; there is a right answer per task family and budget.

Approach Example system Hardware cost Control fidelity Operator ramp Best for Weakness
Bimanual leader-follower ALOHA (Stanford) ~$20k; Mobile ALOHA ~$32k with base Joint-level, 50 Hz, kinesthetic Fast (days) Fine bimanual manipulation Cost per station; fixed embodiment
Low-cost kinematic leader GELLO ~$300 per leader arm + robot Joint-level, high Fast Scaling stations cheaply on arms you own Build/maintenance effort; per-robot tuning
VR headset + controllers Quest 3 rigs; Open-TeleVision style $500 to $3,500 + robot Pose-level via IK, medium-high Medium (1 to 2 weeks) Humanoids, mobile manipulation, immersive tasks IK artifacts, some operator fatigue/sim sickness
6-DoF desktop device 3Dconnexion SpaceMouse $150 to $450 + robot Pose-level, low-medium Slow for contact-rich work Simple pick-place, lab prototyping Slow, unintuitive for dexterous or bimanual tasks
Robot-free handheld UMI gripper Under $1,500 per device Demonstrations without a robot Fast In-the-wild diversity, pre-robot data Embodiment gap; SLAM-dependent pose quality

Three field notes that the spec sheets miss. First, DROID, one of the largest open manipulation datasets (76,000 episodes, 350 hours, 564 scenes), was collected with a Meta Quest 2 controller driving a Franka arm, proof that consumer VR hardware scales to serious datasets (arxiv.org/abs/2403.12945). Second, GELLO is the highest-throughput-per-dollar option if you already own the follower arms, but plan for a part-time technician; printed leaders drift and servos strip. Third, UMI (arxiv.org/abs/2402.10329) is the only approach that collects before your robot ships, which makes it a strong hedge for teams still finalizing hardware.

For a deeper head-to-head, see our companion piece on comparing teleoperation approaches.

Economics: What Teleoperation Data Actually Costs

The true cost of teleoperation data is operator hours multiplied by an all-in rate, divided by usable episode yield, and most budgets get all three numbers wrong. Hardware is the smaller line item; a $20k ALOHA station running two shifts amortizes to under $3 per collection hour over a year. Labor and QA dominate.

Our first-hand benchmarks, consistent across programs we run:

Cost and throughput factor Typical range (our benchmarks)
All-in operator cost (wages, supervision, facility, hardware amortization) $28 to $60 per teleop hour
Novice operator throughput (weeks 1 to 2) 8 to 12 usable episodes per hour
Trained operator throughput (week 3 onward) 25 to 40 usable episodes per hour
QA rejection rate (mature pipeline) 10 to 15 percent
QA rejection rate (new task or new rig, first 2 weeks) 20 to 30 percent
Effective cost per usable episode roughly $0.90 to $6.00 depending on task complexity

Two implications fall out of that table. The operator ramp curve is a 3x throughput difference, so operator retention is a data-cost lever bigger than any hardware decision; churn resets you to novice yield. And QA rejection compounds against you: a 10,000-episode target with 25 percent rejection means collecting 13,300, not 10,000. We cover the bottleneck math in detail in why teleoperation data is the bottleneck in physical AI.

ROI framing for buyers: at $40 per hour all-in and 30 usable episodes per hour, a 50,000-episode dataset costs about $67k in collection labor before QA overhead. Compare that against three to six months of two ML engineers building and debugging an in-house collection operation, and outsourced collection frequently wins on time even when it ties on cost.

From Demonstrations to Policy: The Imitation Learning Pipeline

The imitation learning pipeline is the sequence that converts teleoperated demonstrations into an evaluated robot policy: collect episodes, convert to a training format, train ACT or a diffusion policy, then measure real-world success rate. The pipeline is short; the feedback loop is what teams underuse.

  • Collect and QA. Episodes stream into storage with per-episode metadata (task, operator, rig, success label). QA gates on sync skew, dropped frames, trajectory smoothness, and task success.
  • Convert. HDF5 or ROS 2 bags convert to LeRobot format for training and sharing on Hugging Face.
  • Train. ACT with action chunking (predicting ~100-step action sequences at 50 Hz) or Diffusion Policy for multimodal action distributions.
  • Evaluate. Fixed evaluation protocol: N rollouts per task with randomized initial states, report success rate.
  • Loop. Failure modes from evaluation define next week’s collection priorities.

The proof point worth anchoring on: the original ACT work reached 80 to 90 percent success on fine bimanual tasks such as cable routing and battery insertion from only 50 demonstrations per task (arxiv.org/abs/2304.13705). Demonstration quality and consistency, not raw volume, set that outcome. We have watched 400 sloppy episodes lose to 120 clean ones on the same task, same architecture. Collect less, reject more, retrain sooner.

Case Study Proof: Scaling a VLA Data Program

A humanoid foundation model team came to us needing 12,000 QA-passed bimanual episodes across 40 household tasks in ten weeks, after their in-house effort had produced 1,900 episodes in eight weeks with a 34 percent rejection rate. We stood up six leader-follower stations, ran a two-week operator certification program, and enforced automated sync and smoothness gates at ingest. By week four the floor was producing 1,400 usable episodes per week; the program closed at 12,600 QA-passed episodes with a 12 percent final rejection rate, and the team’s ACT baseline improved from 41 to 68 percent average task success on their internal eval. The full breakdown, including what failed in weeks one and two, is in the VLA case study.

Standing Up a Collection Program: The First 30 Days

A collection program launch is a 30-day sequence of hardware bring-up, protocol authoring, operator certification, and QA gate activation, in that order. Teams that compress or reorder it pay the cost back with interest in weeks five through eight. Our standard sequence:

Days 1 to 7: rigs and clocks. Assemble stations, calibrate cameras (record intrinsics and extrinsics per rig, per calibration), and prove time sync before anyone collects a demonstration. Log a physical sync event (a clapboard works) and verify every stream lands within one camera frame. A day spent here saves a re-collection later; skew discovered after 5,000 episodes is not fixable in post.

Days 8 to 14: protocol. Write task cards for the first five tasks: success criteria, reset procedure, randomization zones marked physically on the workspace, and expected episode duration. Pilot each card yourself before an operator touches it. Every ambiguity you leave in a task card becomes a labeling inconsistency in the dataset.

Days 15 to 28: certification. Run operators through reference tasks with a throughput bar (we use 20 usable episodes per hour) and a quality bar (QA pass rate above 80 percent) before production collection begins. Expect the standard ramp: 8 to 12 usable episodes per hour early, 25 to 40 after two to three weeks.

Day 30: gates on. Nightly automated QA becomes mandatory, with per-operator, per-rig reports every morning. From this point, the program self-corrects.

Build vs Buy: The Honest Decision Criteria

The build-versus-buy question comes down to whether demonstration collection is a competency your team needs to own or a supply problem it needs solved. Building makes sense in three cases: your embodiment is unusual enough that no vendor has matching rigs, your data requirements are small (under roughly 5,000 episodes), or collection know-how is itself part of your product thesis. Buying makes sense when the deadline is the constraint. The math from the economics section applies either way, but in-house programs carry two costs that rarely appear in the comparison spreadsheet: the two to four engineer-months of pipeline and QA tooling before the first reliable episode, and the management overhead of running an operations floor inside an ML organization. We are a vendor, so discount our view accordingly; then ask any team that has run both models which cost they underestimated.

Free Resource: Teleoperation Data RFP Scorecard

A vendor RFP scorecard is a weighted rubric for comparing teleoperation data providers on the criteria that actually predict dataset quality. Ours scores ten dimensions, including sync guarantees, QA rejection transparency, operator training programs, format deliverables, and per-episode pricing disclosure. Most vendors will not publish cost per hour; make them fill in the sheet. [Download the RFP scorecard (XLSX)] and use it on us too.

Related reading: 5 hidden challenges in teleoperation data collection.

Next Step

If you are scoping a demonstration data program, two options. Download 10 free teleoperation episodes (HDF5 plus synchronized MP4, three camera views, 50 Hz actions) and run them through your training stack. Or book a 30-minute scoping call with our data operations team and leave with a per-episode cost estimate for your task list. [Download Sample Data] | [Book a Demo]

Request a Quote

Frequently Asked Questions

What is teleoperation data collection?

Teleoperation data collection is the recording of synchronized robot actions, joint states, and camera observations while a human operator remotely controls the robot through a task. The resulting episodes train imitation learning policies such as ACT and diffusion policies, which underpin VLA models.

Our all-in benchmark is $28 to $60 per operator hour, covering wages, supervision, facility, and hardware amortization. With trained operators producing 25 to 40 usable episodes per hour, effective cost lands between roughly $0.90 and $6.00 per usable episode depending on task complexity.

Fewer than most teams assume, if quality is high. The original ALOHA/ACT results reached 80 to 90 percent success on fine bimanual tasks with about 50 demonstrations per task. Multi-task and VLA training raises the requirement into the thousands per skill family, which is where collection economics start to dominate.

It depends on the task family and budget, per the comparison table above. Fine bimanual manipulation favors leader-follower rigs (ALOHA at ~$20k, or GELLO leaders at ~$300 per arm on robots you own). Humanoid and mobile tasks favor VR teleop. Robot-free UMI capture wins when hardware has not shipped yet.

The main causes are inter-stream time skew above one camera frame, dropped frames, jerky or hesitant trajectories, failed task completions, and reset protocol violations. Expect 20 to 30 percent rejection in the first two weeks on a new task and 10 to 15 percent once the pipeline matures.

Yes. UMI-style handheld grippers record demonstrations with a wrist-mounted camera and SLAM-derived poses, no robot required. The trade-off is an embodiment gap when transferring to your platform, so most teams use it to supplement, not replace, on-robot teleop.

HDF5 per episode and the LeRobot dataset format are the current de facto standards; ROS 2 bags are common at capture time. Whatever the container, require hardware timestamps on every stream and per-episode metadata (task, operator ID, rig ID, success label).

Why Egocentric Data Collection for Robotics Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

In the kitchen bay of our capture floor, a collector claps once in front of her helmet camera, waits for the sync check to pass on the wall monitor, and begins unloading a dishwasher using only grasps a parallel-jaw gripper could copy. By the end of her shift she will have produced more usable first-person manipulation data than one of our bimanual teleoperation stations yields in a week. Nothing about that scene is exotic. The exotic thing is how rare it still is.

It is rare for a structural reason. Text and images were exhaust from human activity; someone else already paid to produce them. First-person manipulation data is not exhaust. Nobody films their hands doing the dishes with calibrated stereo cameras and synchronized IMUs, so every hour has to be manufactured deliberately, by someone wearing a rig or driving a teleoperation station, and then inspected before it is fit to train on. The largest open robot manipulation dataset, Open X-Embodiment, took 21 institutions pooling 22 embodiments to reach roughly a million trajectories (arXiv:2310.08864), while language models trained on trillions of tokens.

This post argues one thesis: the binding constraint on physical AI right now is the supply of QA-passed egocentric hours, not compute and not architecture. You will get the arithmetic behind that claim, our cost benchmarks for the three ways teams try to break the constraint, and a checklist for scaling collection without torching your budget.

We spend our days at DexSet manufacturing exactly this data: egocentric, exocentric, and teleoperation streams for VLA and humanoid teams. The numbers below come from our production logs, not analyst reports.

Key Takeaways

  • Robot learning is data-bound: internet-scale text corpora dwarf the largest robot manipulation datasets by many orders of magnitude, and almost no internet video is first-person with recoverable actions.
  • Teleoperation alone cannot close the gap. A well-run bimanual station yields roughly 15 to 25 usable episode-hours per week; at $28 to $60 per hour, 10,000 hours is a multi-year, seven-figure program.
  • Egocentric human capture shifts the curve: $15 to $40 per hour in our benchmarks, 3 to 5x the collection rate per operator, and it scales with people instead of robot fleets.
  • The constraint is not raw hours, it is QA-passed hours. Plan for 15 to 25 percent of captured footage to fail quality gates.
  • Scaling works when you treat collection as a manufacturing process: fixed protocols, instrumented rigs, sampled QA, and annotation depth matched to the training objective.

Why Data, Not Compute, Is the Binding Constraint

The binding constraint in a training pipeline is the input you cannot buy more of at market price, and for physical AI that input is embodiment-relevant data. Compute is rentable by the hour. Architectures are published within months of working. But there is no marketplace where you can order 10,000 hours of first-person manipulation with hand pose annotations for next quarter, because until recently the industry that manufactures it did not exist.

Look at what the flagship models actually consumed. RT-2 leaned on internet-scale vision-language pretraining precisely because robot data was scarce (arXiv:2307.15818). OpenVLA trained on 970K robot episodes from Open X-Embodiment (arXiv:2406.09246). π0 combined internet-scale pretraining with a proprietary dexterous manipulation corpus (arXiv:2410.24164). The pattern across all three: model capability tracked data supply, and every team supplemented public data with expensive private collection.

Meanwhile the largest first-person video corpus in existence, Ego4D, holds 3,670 hours (arXiv:2110.07058). Impressive for a research effort, and still less footage than a single mid-size YouTube channel uploads in a year. For robotics, most of it lacks the calibration and action annotations that policy learning needs. The shelf is nearly empty. Whoever wants the data has to make it.

The Teleoperation Ceiling

The teleoperation ceiling is the practical throughput limit of collecting robot demonstrations through human-controlled rigs, and it is lower than most roadmaps assume. ALOHA-style bimanual stations (arXiv:2304.13705) made teleop dramatically cheaper to build; the ALOHA rig famously targeted a ~$20K bill of materials against six-figure predecessors. What they did not change is the physics of throughput.

In our teleop operations, a trained operator on a healthy rig produces 15 to 25 usable episode-hours per week once you subtract resets, calibration, failed episodes, and operator fatigue. Push operators harder and quality drops before throughput rises; fine manipulation under fatigue produces exactly the jerky, hesitant trajectories you do not want a policy imitating.

Now run the arithmetic for a 10,000-hour corpus, which is a modest target by foundation model standards:

ApproachCost per QA-passed hour (our benchmarks)Realistic throughputTime to 10,000 hoursProgram cost
Teleoperation, 5 bimanual rigs$28 to $6075 to 125 hrs/week1.5 to 2.5 years$280K to $600K
Teleoperation, 20 rigs$28 to $60300 to 500 hrs/week5 to 8 months$280K to $600K + fleet capex
Egocentric human capture, 20 collectors$15 to $40400 to 700 hrs/week4 to 6 months$150K to $400K

The 20-rig teleop row hides the real problem: rig fleets need robot hardware, lab space, maintenance engineers, and operator pipelines. Twenty humans wearing $500 to $1,200 capture rigs need a protocol document and a QA queue. That asymmetry is the entire argument for egocentric capture as the scaling layer, with teleop reserved for the embodiment-matched fine-tuning set. Systems like EgoMimic have shown the combined recipe outperforms robot data alone (arXiv:2410.24221).

Why Egocentric Capture Is Hard to Scale Anyway

Scaling egocentric collection means turning a research-style capture activity into a manufacturing process with yield metrics, and most attempts stall on the same four problems.

Yield, not volume. Raw hours are easy; QA-passed hours are the product. Across our programs, 15 to 25 percent of captured footage fails gates for motion blur, dropped IMU packets, sync drift, or off-protocol behavior. Teams that do not measure yield discover it at training time, which is the most expensive place to discover it.

Calibration debt. Every uncalibrated GoPro hour you collect today is an hour of 3D lifting problems you inherit later. Devices like Aria carry factory calibration and machine perception services (projectaria.com); consumer rigs need per-unit calibration routines and periodic re-checks, and skipping them is the most common false economy we see.

Annotation as the cost center. Capture at $15 to $22 per hour feels cheap until hand pose and object tracks push finished cost to $30 to $40. The fix is not cheaper annotation; it is refusing to annotate hours your training plan does not need at that depth.

Protocol drift. Thirty collectors interpret “tidy the workspace” thirty ways. Without scripted task cards, fixed start states, and outcome labels, you end up with footage instead of episodes. The LeRobot dataset conventions (github.com/huggingface/lerobot) are a useful forcing function here: if a capture cannot be expressed as episodes with consistent schema, it probably was not collected with enough discipline.

A Scaling Checklist That Survives Contact With Reality

A scaling checklist is only useful if each item maps to a failure we have actually watched happen, so here is ours:

  • Define the training mechanism first (encoder pretraining, retargeting, co-training), then derive modalities and annotation depth from it.
  • Set a QA yield target (we hold vendors, including ourselves, to 80 percent+ first-pass) and sample-inspect every batch.
  • Match camera height and field of view to the deployment embodiment before collecting hour one.
  • Script tasks as episodes: start state, goal, outcome label, retry policy.
  • Calibrate on a schedule, not on failure.
  • Track cost per QA-passed hour weekly; it is the only number that predicts your budget.
  • Pilot 50 to 100 hours and train on them before committing to thousands.

If you want the full treatment of rigs, modalities, and economics behind this checklist, it lives in our pillar guide: The Complete Guide to Egocentric Data Collection for Robotics.

Frequently Asked Questions

Why is data the bottleneck in physical AI rather than compute?

Compute and architectures are purchasable and published, while first-person manipulation data with recoverable actions barely exists on the internet and must be manufactured deliberately. The largest robot datasets hold on the order of a million trajectories, orders of magnitude below the corpora that trained large language models.

In DexSet’s operations, a trained operator on a bimanual rig produces roughly 15 to 25 usable episode-hours per week after resets, failures, and fatigue. Scaling further requires more robot rigs, lab space, and maintenance, which is why teleop fleets scale slowly.

Our benchmarks put QA-passed egocentric capture at $15 to $40 per hour depending on annotation depth, against $28 to $60 per hour for teleoperation, with 3 to 5 times higher collection throughput per operator.

No. The embodiment gap between human hands and robot grippers means policies still need robot-embodiment demonstrations. The strongest published results co-train on both, as in EgoMimic.

Plan for 15 to 25 percent of raw captured footage to fail quality gates for blur, sync drift, sensor dropouts, or protocol violations.

From Bottleneck to Plan

If your roadmap has a data line item and a question mark next to it, start with the pillar guide above, then request a sample batch from us. We ship real episodes with real QA reports, and the pricing table is on the first page, not behind a discovery call.

Request a Quote

Teleop, Egocentric, Exocentric, or Handheld: Comparing Robot Training Data Approaches by Cost

Posted on by Sainath Gupta

In February, we priced the same 5,000-hour manipulation corpus three different ways for one buyer: all bimanual teleoperation at a blended $44 per hour, a 65/35 egocentric-to-teleop mix at just under $30, and a handheld-gripper-heavy plan in between. Same task list, same acceptance spec. The spread between the first two plans came to roughly $70,000.

I build teleop cells for a living, so this is not an argument against teleoperation. It is the thesis that quoting exercise made unavoidable: modality selection is a budgeting decision, and the right modality for each training objective is the cheapest one that actually satisfies it. For plenty of what your model needs to learn, the premium teleop hour is simply the wrong purchase.

Teams get this wrong in both directions. Some buy 20,000 hours of premium teleop and burn budget teaching their encoder what a kitchen looks like, a job $18-per-hour egocentric video does fine. Others go all-in on cheap human video and then discover their policy has beautiful representations and no idea how to move an actual gripper.

The reason the mistake is so common is that modality costs and modality capabilities are usually discussed separately. Cost tables live in procurement decks; capability arguments live in arXiv papers. This post puts them in one place, with DexSet’s operating benchmarks attached, so you can match each dollar to the learning objective it actually serves.

By the end you will have per-hour costs for four collection approaches, an honest pros-and-cons list for each, and a decision matrix that maps training objectives to the cheapest modality that satisfies them.

Key Takeaways

  • Teleoperation ($28 to $60/hr) is the only approach that outputs executable robot actions natively. Pay for it where action supervision matters.
  • Egocentric human video ($15 to $40/hr) is the cheapest volume play, best for representation pretraining, worst for the embodiment gap.
  • Multi-view exocentric capture ($20 to $50/hr) buys scene context and cross-view consistency; calibration labor is its hidden cost.
  • UMI-style handheld grippers (under $1k per device by our estimates) collect gripper-centric data at near-egocentric labor cost, with heavier post-processing.
  • QA rejection (10 to 30 percent in our pipelines) and annotation ($8 to $25/hr per pass) apply to all four. Compare on cost per usable hour.

The Four Approaches, Defined

A data collection approach is the pairing of a capture device with a control source: robot teleoperation, head-mounted egocentric capture, calibrated exocentric camera arrays, or handheld instrumented grippers. Everything else, mono versus stereo, camera count, annotation depth, is a variation within these four.

Teleoperation: $28 to $60 per Hour

Teleoperation data is produced by a human driving a real robot through leader-follower arms or a VR interface, so every recorded frame pairs observations with executable actions in the robot’s own action space. The ALOHA project showed this could be done on a roughly $20k bimanual rig (https://arxiv.org/abs/2304.13705), and Mobile ALOHA extended it to whole-body mobile tasks on roughly $32k of hardware (https://arxiv.org/abs/2401.02117).

Pros

  • Native action labels; feeds imitation learning and VLA post-training directly
  • Matches your exact embodiment, gripper, and camera placement
  • Long-horizon, contact-rich tasks are demonstrable at production quality

Cons

  • Highest labor cost: trained operators at $18 to $38 per hour, plus rig amortization and QA
  • Throughput capped by operator skill and episode reset time
  • Data is embodiment-specific; switching robots strands some of its value

Egocentric Human Video: $15 to $40 per Hour

Egocentric data is first-person video from a head-mounted camera while a human performs tasks with their own hands, capturing human-level dexterity with no robot in the loop. Ego4D and EgoExo4D (https://arxiv.org/abs/2311.18259) made the research case; the commercial case is pure economics, since the collector works at natural speed on tasks they already know.

Pros

  • Cheapest per hour; scales to thousands of hours quickly
  • Enormous task and scene diversity, including real homes
  • Strong pretraining signal for visual encoders and hand-object interaction priors

Cons

  • No robot actions; the embodiment gap means it rarely supervises control directly
  • QA rejection skews high (motion blur, gaze drift, occlusion), 15 to 30 percent in our pipelines
  • Needs retargeting or paired data to transfer to a gripper

Multi-View Exocentric Capture: $20 to $50 per Hour

Exocentric data is third-person video from multiple calibrated, synchronized cameras observing the same task, giving models scene-level context and cross-view consistency that neither ego nor teleop streams provide alone. Stereo pairs add 15 to 25 percent over mono at the same view count and buy metric depth in return.

Pros

  • Full-scene coverage; occlusions in one view are recovered in another
  • Calibrated multi-view supports 3D reconstruction and world-model training
  • Pairs well with egocentric streams (the EgoExo4D recipe)

Cons

  • Calibration and synchronization labor at every scene change is the silent budget eater
  • Fixed arrays limit scene diversity; mobile arrays raise cost
  • Still no action labels without a paired control source

Handheld Instrumented Grippers (UMI-style): Near-Egocentric Cost, Gripper-Centric Output

Handheld gripper capture uses a portable, wrist-camera-equipped gripper operated by a human, producing gripper-centric trajectories without any robot present at collection time. The UMI paper (https://arxiv.org/abs/2402.10329) defined the category; our build estimate is under $1,000 per device including the camera.

Pros

  • Capex is trivial next to a $20k to $32k teleop cell
  • Collection happens anywhere a person can walk, at near-egocentric labor rates
  • Output is closer to robot action space than raw human video

Cons

  • Heavier post-processing to recover clean actions (SLAM drift, kinematic mismatch)
  • Gripper form factor constrains which tasks are demonstrable
  • QA tooling for this modality is younger; expect iteration

Master Comparison Table

Approach DexSet cost/raw hr Capex per station Action labels QA rejection Best use
Teleoperation $28 to $60 $20k to $32k Native 10 to 25% VLA post-training, imitation learning
Egocentric video $15 to $40 $300 to $3.5k None 15 to 30% Encoder pretraining, dexterity priors
Exocentric multi-view $20 to $50 $5k to $15k (3 to 8 cams) None 10 to 20% Scene context, world models, 3D
Handheld gripper (UMI-style) $18 to $42 (our estimate) Under $1k/device Recoverable 15 to 25% Diverse-scene manipulation at low capex

Annotation is additive to every row: $8 to $12 per hour for language instructions, up to $18 to $25 for dense masks and contact labels. And every row’s real price is its cost per usable hour: divide by (1 minus rejection rate). The full math, with a budget spreadsheet, lives in our robot training data costs and pricing guide.

Decision Matrix: Match the Dollar to the Objective

A modality decision matrix assigns each training objective the cheapest approach that actually satisfies it, instead of defaulting everything to the premium modality. Here is the one we use in scoping calls:

Your objective Buy this Not this Why
Post-train a VLA on your robot Teleoperation Egocentric You need native actions on your embodiment
Pretrain visual encoders at volume Egocentric Teleoperation Paying $42/hr for pixels is waste
Scene diversity across 100+ homes Handheld gripper or egocentric Fixed exo array Portability beats calibration
Depth-dependent manipulation Stereo exo + teleop Mono anything Metric depth earns its 15 to 25% premium
World-model or video-prediction training Exo multi-view + ego pairs Teleop only Cross-view consistency is the signal
Bimanual, contact-rich skills Teleoperation (ALOHA-class) Handheld gripper Two grippers, force-aware demos

The pattern behind the matrix: mix modalities and stage them. Open X-Embodiment’s 1M+ trajectories across 22 embodiments (https://arxiv.org/abs/2310.08864) and DROID’s 76k episodes (https://arxiv.org/abs/2403.12945) already prove cross-source data mixes train better generalists. Your budget should look like a portfolio, not a single line item. A 70/30 split of cheap pretraining hours to teleop post-training hours routinely cuts blended cost by a third in programs we run, with no loss on the action-supervised objectives.

Match the Modality to the Objective

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the cheapest way to collect robot training data?

Egocentric human video, at $15 to $40 per hour in our benchmarks, since hardware is a wearable camera and collectors work at natural speed. It is cheapest per hour but supplies no robot actions, so it cannot carry a program alone.

When you need executable actions on your specific embodiment: VLA post-training, imitation learning for contact-rich or bimanual skills. For those objectives nothing cheaper substitutes, which is exactly why you should not spend teleop dollars on anything else.

They are production-useful with caveats. Capex is under $1,000 per device by our estimates and scene diversity is the widest of the four approaches, but plan for heavier post-processing and a 15 to 25 percent rejection rate while the QA tooling matures.

Stereo adds 15 to 25 percent to exocentric capture cost and pays for itself on depth-dependent manipulation tasks. For pure representation pretraining, mono volume usually beats stereo precision per dollar.

Convert everything to cost per usable hour: quoted rate divided by (1 minus the measured QA rejection rate), plus annotation per pass. Our pricing guide includes the worked tables.

Start with the Robot Training Data Costs and Pricing Guide for the full benchmark tables, then book a scoping call and we will run your task list through the decision matrix above.

Robot Training Data Costs and Pricing: The Complete 2026 Guide

Posted on by Sainath Gupta

In our first year we priced a 6,000-hour teleoperation program the way most of this market still prices: by the raw hour. We hit our quoted rate and still broke the client’s budget model, because 22 percent of captured episodes failed their acceptance spec and nobody’s plan had funded the recollection. We rebuilt our cost model around that miss, and this guide is the rebuilt model, published.

The mistake was possible because robot training data has no commodity unit yet. An “hour” of data can mean a raw teleop stream with 30 percent unusable episodes, or a QA-passed, annotated, deduplicated hour that trains a policy. Those two hours differ in cost by 2x or more, and vendors quietly quote whichever one makes their number look better.

So here is the thesis this guide argues from the first table to the last: the only honest unit for pricing robot training data is cost per usable hour, and every quote you receive should be converted into that unit before you compare anything. To make the conversion possible, you will get our first-hand cost-per-hour benchmarks for teleoperation, egocentric human video, and multi-view exocentric capture; rig capex figures anchored to public hardware like ALOHA and UMI; the QA rejection math that separates raw hours from usable hours; and a worked budget for a 10,000-hour VLA data program.

DexSet collects egocentric, exocentric, teleoperation, mono, and stereo data for physical AI teams. We run these rigs, staff these operators, and eat these QA rejections every week. Every number below is either our own operating benchmark or a cited public source.

TL;DR: Robot Training Data Costs at a Glance

  • Teleoperation data: $28 to $60 per raw hour (rig amortization + operator + QA), based on DexSet benchmarks.
  • Egocentric human video: $15 to $40 per hour, the cheapest scalable modality.
  • Multi-view exocentric capture: $20 to $50 per hour depending on camera count and calibration load.
  • Annotation passes: $8 to $25 per hour extra, on top of any capture modality.
  • Rig capex: ~$20k for an ALOHA-style bimanual station (per the ALOHA paper), ~$32k for Mobile ALOHA, under $1k per UMI-style handheld gripper by our build estimates.
  • QA rejection runs 10 to 30 percent in our pipelines, so always budget on cost per usable hour, not raw hour.
  • Public scale references: Open X-Embodiment aggregates 1M+ trajectories across 22 embodiments; DROID contains 76k episodes.

What Do Robot Training Data Costs Actually Include?

Robot training data cost is the fully loaded price of producing one hour of demonstration data that a robot learning pipeline can actually consume, covering hardware amortization, operator or collector labor, QA review, annotation, and delivery infrastructure. Most published debates skip half of these line items, which is why budgets built from a single “per hour” quote fall apart in month two.

A defensible cost model has five layers:

  • Capture hardware (capex). Teleop stations, headsets, camera arrays, grippers. Amortized over 12 to 24 months of use.
  • Collection labor (opex). Teleoperators, camera-wearing collectors, scene setup staff. Usually the largest single line.
  • QA and curation (opex). Human review, automated checks, rejection and recollection. The most commonly omitted layer.
  • Annotation (opex). Language instructions, segmentation, keyframes, success labels. Priced per pass.
  • Infrastructure (opex). Storage, upload bandwidth, format conversion, dataset versioning.

When a vendor quotes you a single number, ask which of these five layers it covers. In our experience the quoted number usually covers layers 1 and 2 and quietly excludes 3 through 5, which add 30 to 80 percent on top.

Core Modalities and What Each One Costs

A data modality is the combination of sensor viewpoint and control method used to produce demonstrations: teleoperation, egocentric human video, exocentric multi-view capture, and their mono or stereo variants. Each modality has a distinct cost structure because each one shifts spend between hardware, labor, and QA differently.

Teleoperation Data: $28 to $60 per Hour

Teleoperation data is demonstration data produced by a human directly controlling a robot, typically through a leader-follower arm setup or a VR interface, so the recorded actions are executable robot trajectories. It is the gold standard for imitation learning and VLA post-training because actions come out in the robot’s own action space, but it is also the most expensive modality per hour.

Our benchmarks across bimanual manipulation programs:

  • Rig amortization: $4 to $9 per hour (a $20k to $32k station amortized over 18 months of two-shift use, including maintenance and spare grippers).
  • Operator labor: $18 to $38 per hour depending on region, task dexterity, and whether the task needs trained specialists (cable routing and garment handling sit at the top of that range).
  • QA overhead: $6 to $12 per hour, covering episode review, rejection, and partial recollection.

Total: $28 to $60 per raw teleop hour. Long-horizon mobile manipulation lands at the top of the range; tabletop pick-and-place with experienced operators lands at the bottom.

The entity chain matters here: teleoperation feeds imitation learning methods like ACT, which the ALOHA project introduced, and imitation learning at scale is what current VLA models are built on. The ALOHA paper (Zhao et al., “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” https://arxiv.org/abs/2304.13705) demonstrated that a roughly $20k bimanual rig could produce data good enough for fine manipulation, which reset industry assumptions about capture hardware pricing.

Egocentric Human Video: $15 to $40 per Hour

Egocentric data is first-person video captured from head-mounted or body-mounted cameras while a human performs tasks with their own hands, giving models human-level dexterity examples without any robot in the loop. It is the cheapest modality to scale because the “rig” is a wearable and the collector is doing a task they already know how to do.

Our cost structure:

  • Hardware amortization: $1 to $4 per hour. Headsets and head-mounted cameras cost $300 to $3,500 and survive thousands of capture hours.
  • Collector labor: $10 to $26 per hour.
  • QA overhead: $5 to $10 per hour. Egocentric QA is dominated by motion blur, gaze drift, and occlusion checks.

Total: $15 to $40 per hour. The catch is the embodiment gap: human hands are not robot grippers, so egocentric data usually pretrains representations rather than directly supervising actions. Datasets like Ego4D and EgoExo4D (https://arxiv.org/abs/2311.18259) established the research value of this modality; the commercial question is purely about cost-effective volume.

Multi-View Exocentric Capture: $20 to $50 per Hour

Exocentric data is third-person video captured from fixed or mobile external cameras observing a task from multiple calibrated viewpoints, which gives models scene context and cross-view consistency that a single egocentric stream cannot. Cost scales with camera count and, more painfully, with calibration and synchronization labor.

Our benchmarks: $20 to $50 per hour for 3 to 8 synchronized views, including calibration checks at every scene change. Stereo pairs add roughly 15 to 25 percent over mono at the same view count because of the extra calibration and QA burden, but they buy you metric depth, which matters for manipulation policies.

Annotation: $8 to $25 per Hour, on Top of Everything Above

Annotation cost is the incremental price of adding structured labels to captured data: language instructions, subtask segmentation, success and failure flags, object masks, or keyframe tags. It is always additive to capture cost, and it is where “cheap” datasets quietly become expensive.

Typical per-pass pricing from our pipelines:

  • Language instruction labeling: $8 to $12 per data hour
  • Subtask segmentation: $10 to $16 per data hour
  • Dense object masks or contact annotation: $18 to $25 per data hour

Comparison Table: Cost per Hour by Modality

Modality Raw cost/hour (DexSet benchmark) Hardware amortization share QA rejection rate Best suited for
Teleoperation (bimanual) $28 to $60 $4 to $9 10 to 25% VLA post-training, imitation learning
Egocentric human video $15 to $40 $1 to $4 15 to 30% Representation pretraining, hand priors
Exocentric multi-view (3 to 8 cams) $20 to $50 $3 to $7 10 to 20% Scene understanding, cross-view learning
Stereo add-on (vs mono) +15 to 25% +$1 to $2 +2 to 5 pts Depth-dependent manipulation
Annotation pass (language) +$8 to $12 n/a n/a Instruction-following VLAs
Annotation pass (dense masks) +$18 to $25 n/a n/a Grasp and contact modeling

Rig Economics: Capex Benchmarks You Can Verify

Rig capex is the upfront hardware cost of a capture station before a single hour of data exists, and it is the number that determines whether building in-house ever beats buying data. The public record here is unusually good, so you do not have to trust vendor hand-waving.

Rig Approx. capex Source Notes
ALOHA bimanual teleop station ~$20,000 ALOHA paper, https://arxiv.org/abs/2304.13705 Leader-follower arms, cameras, frame
Mobile ALOHA ~$32,000 Mobile ALOHA paper, https://arxiv.org/abs/2401.02117 Adds mobile base for whole-body tasks
DexSet production teleop cell $20,000 to $32,000 First-hand DexSet build costs ALOHA-class arms plus industrial cameras, lighting, sync hardware
UMI handheld gripper Under $1,000 per unit (our build estimate) UMI paper, https://arxiv.org/abs/2402.10329 Portable gripper with wrist camera; no robot needed at capture time
Egocentric headset kit $300 to $3,500 Consumer/enterprise hardware pricing Camera glasses to mixed-reality headsets

Two practical lessons from running these rigs:

First, capex is rarely the problem. A $26k teleop cell running two shifts amortizes to under $9 per hour within 18 months. Labor and QA dominate every mature program we run.

Second, UMI-style handheld grippers changed the low end of the market. Because the capture device is a portable gripper with a wrist camera rather than a full robot cell, collection can happen in real homes and kitchens at egocentric-like labor costs while still producing gripper-centric trajectories. The trade-off is a heavier post-processing and QA load to recover clean actions.

Cost per Usable Hour: The Number That Actually Matters

Cost per usable hour is the total program spend divided by the hours that survive quality assurance, and it is always higher than the quoted cost per raw hour. This is the single most important correction to apply to any vendor quote, including ours.

In DexSet pipelines, QA rejection runs 10 to 30 percent depending on modality and task difficulty. Episodes get rejected for dropped frames, desynchronized views, failed task completion, occluded end-effectors, or annotation mismatches. The math is unforgiving:

Usable-hour math. At $40 per raw teleop hour with a 25 percent rejection rate, your real cost is $40 / 0.75 = $53.33 per usable hour. A competitor quoting $36 per hour with an unmeasured 35 percent rejection rate is actually charging $55.38. The cheaper quote is the more expensive dataset.

Quoted raw $/hr Rejection rate True cost per usable hour
$30 10% $33.33
$30 30% $42.86
$40 15% $47.06
$40 25% $53.33
$55 10% $61.11

When you evaluate any provider, require three things in writing: the measured rejection rate on a comparable program, who pays for recollection of rejected episodes, and whether QA review labor is inside or outside the quoted rate. If a vendor cannot produce a rejection rate, they are not measuring quality.

Budgeting a Program: A Worked Example

A data budget is a forward plan that converts a target usable-hour count into total spend across capture, QA, annotation, and infrastructure. Here is a realistic model for a VLA team that needs 10,000 usable teleop hours with language annotation.

  • Target: 10,000 usable hours
  • Assumed rejection rate: 20 percent, so raw capture target = 12,500 hours
  • Blended teleop rate: $42 per raw hour = $525,000 capture
  • Language annotation at $10 per usable hour = $100,000
  • Storage, versioning, delivery at roughly 4 percent of capture = $21,000
  • Total: ~$646,000, or $64.60 per usable annotated hour

For scale context, Open X-Embodiment pooled more than 1 million trajectories across 22 robot embodiments from 21 institutions (https://arxiv.org/abs/2310.08864), and DROID contains 76,000 episodes collected across 52 buildings (https://arxiv.org/abs/2403.12945). Those datasets exist because no single lab could afford to collect that volume alone, which tells you what the market already knows: collection cost, not model architecture, is the binding constraint on physical AI progress.

Build vs Buy: When Each One Wins

The build-vs-buy decision compares the fully loaded cost of standing up your own capture operation against a vendor’s cost per usable hour at your required volume and quality bar. Neither answer is always right; the crossover depends on volume, duration, and how much operational pain you can absorb.

Build wins when you need under roughly 2,000 hours of highly proprietary, robot-specific data, you already own the robots, and engineering time is genuinely available. Buy wins when you need volume and velocity: a vendor already amortized the rigs, trained the operators past the learning curve (operator throughput improves 30 to 50 percent over their first 200 hours in our programs), and built the QA tooling you would otherwise write from scratch. Most funded teams land on a hybrid: build one internal cell for rapid task iteration, buy production volume.

Case Study Proof: A Humanoid Foundation Model Team

A humanoid foundation model team came to us with a $400k data budget, a quoted competitor rate of $35 per hour, and a plan for 11,400 hours. The quote excluded QA review and carried no measured rejection rate. On a 200-hour pilot we measured 28 percent rejection against their own spec, which repriced the competitor dataset at $48.60 per usable hour before annotation.

We restructured the program: tightened the task spec to cut ambiguity-driven rejections, moved 30 percent of volume to egocentric capture for representation pretraining, and reserved teleop for post-training data. Result: 9,800 usable hours delivered inside the original budget, with rejection stabilized at 12 percent by week six. The lesson is not that our rate was lower. It is that cost per usable hour, measured on a pilot, is the only number that predicted their final spend.

Why Most Vendors Hide Pricing, and Why We Publish It

Hidden pricing is a deliberate market structure in which vendors quote deal by deal to maximize price discrimination, and it survives because buyers lack a shared benchmark. Large annotation-era incumbents built their margins on this asymmetry, and robot data inherited the habit.

We publish our ranges because the buyers we want, Heads of Data who run pilots and measure rejection rates, are exactly the buyers opaque pricing repels. Transparent ranges cost us the occasional overpriced deal and win us every buyer who has been burned before. You should treat any vendor’s refusal to publish even a range as information about how they expect the negotiation to go.

Free Download: The Robot Data Cost Model and RFP Scorecard

We packaged the math in this guide into two working documents: a cost model spreadsheet with editable assumptions for rejection rate, shift count, and amortization period, and a 24-question RFP scorecard covering the five cost layers, QA measurement, and recollection liability. Both are free, no email gate on the scorecard.

Related Reading

Put the Numbers to Work

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

How much does robot training data cost per hour?

Based on DexSet’s operating benchmarks: teleoperation runs $28 to $60 per raw hour, egocentric human video $15 to $40, and multi-view exocentric capture $20 to $50. Annotation adds $8 to $25 per hour per pass. Divide any quoted rate by (1 minus the rejection rate) to get the true cost per usable hour.

The original ALOHA paper reports a bimanual rig built for roughly $20,000, and Mobile ALOHA extends it to whole-body mobile manipulation at roughly $32,000. Our production cells, with industrial cameras, lighting, and sync hardware added, land between $20,000 and $32,000.

In our pipelines, 10 to 30 percent of raw episodes fail QA, depending on modality and task complexity. Well-specified tabletop teleop can hold near 10 percent; long-horizon mobile tasks and loosely specified egocentric capture push toward 30 percent.

Building tends to win below roughly 2,000 hours of proprietary, robot-specific data when you already own robots and engineering time. Buying wins at volume because vendors have amortized rigs, trained operators, and existing QA tooling. Most teams run a hybrid.

Egocentric capture uses wearable cameras and human hands, so hardware costs hundreds to a few thousand dollars and collectors perform familiar tasks at natural speed. Teleoperation requires a $20k to $32k rig plus a trained operator, and outputs executable robot actions, which is what you pay the premium for.

Open X-Embodiment aggregates more than 1 million trajectories across 22 robot embodiments, and DROID contains 76,000 episodes. Both are useful pretraining anchors, but most teams still need proprietary data matched to their own embodiment and tasks.

Require the measured QA rejection rate on a comparable program, clarity on who pays for recollection, an itemized list of which cost layers the rate includes (hardware, labor, QA, annotation, infrastructure), and a paid pilot with your acceptance spec before any volume commitment.

Download the DexSet Robot Data Cost Model and RFP Scorecard, or book a 30-minute pricing walkthrough with our data operations team. We will run your task list through the same model we use internally and hand you the spreadsheet.

Case Study: How We Scaled Exocentric & Multi-View Data for a VLA Model

Posted on by Enoch Pakanati

We once shipped a pilot batch with an 18 millisecond wrist-camera clock offset, and this case study exists partly because of it. The mistake was ours: the wrist stream was timestamped off the arm controller instead of the rig’s PTP clock domain, invisible in playback, and it was the customer’s smoke-test training run at hour 20 that surfaced it. We fixed it, changed our rig standard, and kept the lesson. Vendor case studies that report zero mistakes are describing a project that never touched hardware.

This one covers eight weeks of work with a humanoid foundation model team (anonymized by agreement) whose manipulation policy had stalled on cluttered-scene tasks. The claim we will argue with the numbers below: their ceiling was informational, not architectural, and a disciplined multi-view capture operation, pilot batches, per-session calibration gates, one shared clock domain, is what removed it.

The team arrived with a specific complaint. Their policy trained fine, evaluated fine on open scenes, then dropped hard on cluttered tabletops. Their dataset: tens of thousands of teleoperated episodes, every one recorded from a single fixed camera. Failure review showed what you would expect. When clutter occluded the target from that one viewpoint during approach, the policy guessed.

They asked us for multi-view capture at a rate their internal rig could not hit: hundreds of hours within a quarter, calibrated, synchronized, and QA’d to a standard their ML leads could defend. What follows is how we built it, what it cost, what broke, and what the retrain showed. For the general framework behind these choices, the pillar guide is here: The Complete Guide to Exocentric & Multi-View Data for Robot Learning.

Key Takeaways – 600 hours of DROID-style multi-view episodes (two external stereo + wrist) delivered across 8 weeks on 4 parallel stations. – Per-session calibration gates rejected 4.1 percent of sessions; every rejection would have shipped corrupted extrinsics without the gate. – All-in capture cost landed at $31 per hour, inside our standard $26 to $38 benchmark range. – Retrained on matched episode counts, the customer’s occlusion-heavy split improved by double digits; open-scene performance held.

The Starting Point: Diagnosing a Data Ceiling

A data ceiling is a performance plateau caused by information missing from the training set rather than by model capacity. Confirming one is cheap and worth doing before any capture contract: tag evaluation failures by whether the target object was visible to the training viewpoint at decision time. On this team’s cluttered-scene split, occlusion-correlated failures dominated by a wide margin. No architecture sweep fixes pixels that were never captured.

The finding matches the public record. The RoboMimic study showed observation space design materially changes imitation outcomes on identical demonstrations (arXiv:2108.03298), and DROID’s authors considered multi-view important enough to mandate two calibrated external stereo views plus wrist across all 76,000 episodes (arXiv:2403.12945). We proposed the same geometry rather than inventing one.

The Rig Spec: Boring by Design

The capture spec is the contract between operations and the ML team, and ours fit on one page. Four identical stations, each with:

  • Two ZED 2i stereo cameras on rigid tripod mounts at roughly 45 degrees off the workspace centerline, 1 m from task center, opposite sides.
  • One wrist camera on the arm.
  • Extrinsics calibrated with a ChArUco target via OpenCV, cross-checked in Kalibr; camera-to-robot-base transform verified against known end-effector poses.
  • Sync via PTP-disciplined clocks, with a hard gate: cross-camera skew under 10 ms or the session does not ship.
  • A 20-second ChArUco verification sweep at every session start; reprojection error above 0.5 px blocks capture until recalibration.

Identical stations mattered more than any individual choice. One geometry means one calibration procedure, one QA script, one training data schema, and operators who can rotate between stations without retraining.

What Eight Weeks Actually Looked Like

Scaling capture is a throughput problem with a quality constraint, and the numbers tell the story better than prose:

Metric Value
Capture stations 4 (identical DROID-style geometry)
Calendar time 8 weeks
Delivered episodes ~58,000 across 41 task variants
Delivered hours (multi-view) 600
Sessions rejected at calibration/sync gate 4.1%
Episodes rejected at QA review 2.7%
Storage delivered ~38 TB (H.265, with per-frame extrinsics and sync metadata)
All-in operated cost $31 / hr

Weeks one and two ran at half throughput on purpose. We shipped a 20-hour pilot batch first so the customer’s ML team could confirm schema, load episodes into their LeRobot-based training stack, and run a smoke-test train before we committed the fleet. They caught the wrist-camera clock defect described at the top of this post: a consistent 18 ms offset from timestamping off the arm controller instead of the PTP domain. Catching that at hour 20 instead of hour 600 is the entire argument for pilot batches.

Throughput past the pilot came down to two decisions that had nothing to do with cameras. The first was task design: the customer’s 41 task variants were sequenced so that each station ran one object set per half-day block, which cut resets and scene changes to minutes instead of the constant churn you get when operators bounce between tasks. The second was operator rotation. Because all four stations shared one geometry and one procedure, any operator could run any station, and we scheduled captures to keep stations busy through breaks and calibration stops. Utilization across the fleet held near 85 percent of scheduled hours; on prior projects with heterogeneous rigs, we had struggled to hold 65.

The other failure worth naming: in week five, a boom mount on station three sagged after a fixture swap, and the morning verification sweep caught reprojection error at 1.3 px. The gate rejected the session, recalibration took 25 minutes, and no corrupted data shipped. Before we ran per-session gates, that class of drift used to surface weeks later as unexplainable training noise.

The Result: What the Retrain Showed

The customer retrained the same architecture on matched episode counts, single-view versus our multi-view data, which is the only comparison that isolates the data effect. On their occlusion-heavy cluttered split, success improved by double digits. On open scenes, performance held flat, confirming the ceiling had been informational, not architectural. Their engineers also reported a second-order win: with three calibrated views per episode, failure triage became visual inspection instead of guesswork, because someone could always see what happened.

We report ranges rather than their exact internal metrics by agreement, and we would flag any vendor who publishes a client’s precise evaluation numbers as a reason to negotiate confidentiality carefully. What we can say precisely is what the comparison controlled for: same architecture, same hyperparameters, same episode count, same evaluation protocol. The only variable was the data.

What We Would Repeat, and What We Changed

Three practices carried the project and are now standard on every DexSet engagement: pilot batches before fleet commitment, per-session calibration and sync gates with hard thresholds, and extrinsics embedded in every episode’s metadata rather than in a side document that drifts out of date.

One thing we changed afterward: we now put the wrist camera on the PTP domain from day zero, on every rig, because the 18 ms lesson generalizes. And we stopped quoting capture programs without a failure-tagging pass on the customer’s existing evaluation data first; twice since, that pass showed the bottleneck was not viewpoint at all, and we said so.

Test the Pattern on Your Own Failures

If your evaluation failures cluster around occlusion and your dataset is single-view, the pattern in this case study probably applies to you. Book a demo and we will walk through the pipeline with real sample episodes, calibration metadata included, and run the failure-tagging pass on your evaluation data before anyone talks about a contract.

Request a Quote

Frequently Asked Questions

How long does it take to scale a multi-view capture program?

In this engagement, 600 hours of calibrated multi-view data took 8 weeks on 4 parallel stations, including a deliberately slow 2-week pilot phase. Throughput scales roughly linearly with identical stations once the geometry and QA gates are standardized.

All-in operated capture landed at $31 per hour, inside DexSet’s standard $26 to $38 benchmark range for DROID-style rigs, plus one-time rig builds in the $4,500 to $7,000 range per station.

A 20-second ChArUco verification sweep at every session start, gated on reprojection error under 0.5 px, with extrinsics cross-checked in Kalibr and camera-to-base transforms verified against known end-effector poses. Sessions failing the gate are recalibrated before any capture ships.

Yes. Retrained on matched episode counts, the customer’s occlusion-heavy evaluation split improved by double digits while open-scene performance held, isolating viewpoint coverage as the binding constraint.

Because schema and sync defects are cheap at hour 20 and expensive at hour 600. The pilot here caught a wrist-camera clock offset of 18 ms that would otherwise have contaminated the full delivery.

5 Hidden Challenges in Robotics Data RFPs (and How to Solve Them)

Posted on by Sainath Gupta

The contract was one signature away when the buyer’s counsel asked who owned models trained on the delivered episodes, and the call went quiet for a long moment. Nobody had put the question in the RFP. The clause that eventually surfaced in the vendor’s MSA granted reuse rights over “aggregated and derived data”, which is to say, over the buyer’s task distribution. That deal survived. The assumption that the important questions were already in the document did not.

The obvious RFP challenges are the ones buyers plan for: price, volume, timeline. Those rarely sink a data program. What sinks programs are the questions that never make it into the document, because the buyer did not know the failure mode existed until it arrived inside a delivery.

I run teleoperation rigs for a living, which means I also read the RFPs that reach us. The pattern is consistent: the sections buyers write carefully (pricing tables, volume schedules) cover the risks they have already survived, and the sections they skip (sync tolerances, consent chains, schema versioning) cover the risks they have not met yet. An RFP is a map of its author’s scar tissue.

This post covers the five challenges we most often see missing, why each one stays hidden until it costs money, and the specific question or clause that fixes it. All five fixes are built into the templates in the Robotics Data Buyer’s Playbook.

Key Takeaways

  • The five hidden RFP challenges: undefined sync and calibration specs, licensing and reuse traps, throughput fiction, missing consent chains, and format drift between deliveries.
  • Each has a one-question or one-clause fix that costs nothing to include and is expensive to omit.
  • The common thread: RFPs fail on what they do not ask. Vendors answer the questions in the document, not the ones you meant.

Challenge 1: The Sync Spec That Nobody Defines

A sync spec is the maximum allowed time offset between data streams (cameras, joint states, actions) in a delivered episode, and most RFPs never state one. The omission stays hidden because unsynced data plays back fine to a human eye. It surfaces months later as an imitation learning policy that plateaus below expectations, because the state-action pairs it learned from were misaligned by tens of milliseconds.

The fix: state a tolerance and demand measurement. Our recommended RFP language: “Maximum time-sync error across all streams shall be 10 ms; vendor shall describe how sync is measured, how often it is rechecked during production, and how violations are flagged in delivered metadata.” A vendor who has never measured their sync error will reveal it in one sentence, which is exactly what you paid the question to find out. The same applies to calibration: require per-rig intrinsics and extrinsics files with every delivery and a stated recalibration cadence.

Challenge 2: The Licensing Trap in the MSA

The licensing trap is contract language granting the vendor rights to reuse your delivered episodes, your task definitions, or data derived from them for other customers or their own models. It hides in the master service agreement rather than the RFP response, usually as one clause about “aggregated or derived data.” Your task list and environment setups are competitive information; a reuse clause quietly donates them to whoever buys from the same vendor next.

The fix: ask it in the RFP, before the MSA exists: “Who owns delivered data and models trained on it? Do you retain any right to reuse our episodes, task definitions, or derivatives for any other purpose?” Then score the answer on your vendor scorecard and treat broad reuse rights as disqualifying. Negotiating this after signature means bargaining from a position you already gave away.

Challenge 3: Throughput Fiction

Throughput fiction is a capacity claim (“we can scale to any volume”) unsupported by the arithmetic that produces real hours: rig count times shifts times operators times usable-hour yield. It stays hidden because it fails late, four or six weeks into a ramp, when a vendor sized for 40 hours a week is contractually committed to 150 and quality starts absorbing the difference.

The fix: make the multiplication mandatory. RFP question: “For our program spec, state the number of rigs and operators you would dedicate, the sustained hours per week that supports, and your actual delivered volume for your largest program in the last six months.” Any credible operation knows these numbers instantly. On teleoperation specifically, sustained real-world throughput per bimanual rig runs far below the naive shift math once you account for resets, calibration checks, operator breaks, and QA rejections; in our own operations, a single ALOHA-class rig sustains roughly 20 to 30 usable hours per week, not the 60 to 80 the shift calendar implies.

Challenge 4: The Missing Consent Chain

A consent chain is the documented link between every delivered episode and the signed consent of every human in it: teleoperators, egocentric camera wearers, and bystanders captured in frame. RFPs skip it because it feels like legal boilerplate. It becomes real the day your model ships in a product and your counsel asks whether the training data included identifiable people who never agreed to it, a question with GDPR and biometric-law consequences you cannot retroactively fix.

The fix: two RFP questions. “Describe what operators and captured participants sign, and how consent records map to delivered episodes.” And: “What is your process for faces and bystanders in egocentric capture: avoidance, blurring, or documented consent?” Public egocentric corpora set the reference point here; Ego4D and EgoExo4D shipped with documented consent and privacy processes (arxiv.org/abs/2110.07058; arxiv.org/abs/2311.18259), and a commercial vendor should clear the bar academic datasets already cleared.

Challenge 5: Format Drift Between Deliveries

Format drift is schema change between deliveries from the same vendor: a renamed key in an HDF5 file, a reordered camera list, a new compression setting, an action space silently rescaled. Single-delivery evaluation cannot catch it by definition, which is why it hides through every pilot and surfaces as a broken training pipeline at 2 a.m. before a deadline. The ecosystem’s convergence on versioned formats (LeRobot’s dataset versions, RLDS episode specs, MCAP channel schemas; github.com/huggingface/lerobot, github.com/google-research/rlds, github.com/foxglove/mcap) exists precisely because ad hoc schemas drift.

The fix: contract the schema, not just the format. Require a written schema document as an RFP deliverable, version it as a contract appendix, and add the clause: “Schema changes require written notice one delivery cycle in advance; unannounced schema changes constitute delivery failure.” Then validate mechanically: a loader script in CI that runs on every delivery costs a day to write and catches drift while it is still the vendor’s problem.

Why These Five Stay Hidden

The common mechanism behind all five challenges is delayed feedback: each failure surfaces weeks or months after the decision that caused it, in a different team’s backlog. Sync defects appear as ML debugging tickets. Licensing traps appear in legal review of a partnership, a year later. Throughput fiction appears as a slipped training milestone that gets blamed on the schedule, not the contract. Because the pain lands far from the RFP, the RFP never learns. The fix is not vigilance, which does not scale; it is putting the five questions into a template so they get asked by default, on every deal, including the ones that feel too small or too friendly to need them. Friendly deals with partner labs are where we see the format-drift and consent gaps most often, precisely because nobody wanted to send paperwork to a friend.

The Five Challenges at a Glance

Hidden challenge Why it stays hidden Cost when it surfaces The one-line fix
Undefined sync/calibration spec Bad sync looks fine in playback Weeks of training debugging Require ≤10 ms tolerance, measured and rechecked
Licensing reuse trap Lives in the MSA, not the RFP Task distribution leaks to competitors Ask ownership and reuse questions in the RFP; disqualify broad reuse
Throughput fiction Fails weeks into ramp, not at signing Missed training milestones Demand rigs × shifts × yield arithmetic plus 6-month delivery history
Missing consent chain Feels like boilerplate until launch Legal exposure you cannot backfill Require consent-to-episode mapping and a bystander policy
Format drift Invisible in any single delivery Broken pipelines, silent data corruption Version the schema in the contract; validate every delivery in CI

Close the Gaps Before You Send

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the most expensive hidden challenge in robotics data RFPs?

Format drift and undefined sync specs compete for the title. Sync problems corrupt what the model learns; format drift breaks the pipeline that feeds it. Both are cheap to prevent with one RFP requirement and expensive to diagnose after delivery.

Ask for the arithmetic: dedicated rigs, operators, shifts, and usable-hour yield, plus actual delivered volume on their largest recent program. As a sanity check, a bimanual teleop rig sustains roughly 20 to 30 usable hours per week in real operations.

Any retained right to reuse your episodes, task definitions, or derivatives for other customers or their own models. Ownership of delivered data and models trained on it should sit with the buyer, stated in the RFP response before the MSA stage.

Yes. Teleoperators are identifiable humans generating biometric-adjacent data, and egocentric capture routinely includes bystanders. Require consent records that map to delivered episodes, matching the standard public datasets like Ego4D already meet.

The Complete Guide to Egocentric Data Collection for Robotics (2026)

Posted on by Sainath Gupta

3,670 hours. That is the complete Ego4D corpus, the largest first-person video dataset ever assembled (arXiv:2110.07058), and it amounts to roughly seven months of one person’s waking life. The robot foundation models expected to generalize across every kitchen, warehouse, and workbench on earth are drawing from a first-person data supply about that size, while their language-model cousins trained on trillions of tokens. The shelf is not thin. It is nearly bare, and teleoperation refills it at a few hundred action-labeled hours per rig per year.

The gap exists because robots perceive the world from their own body. A vision-language-action (VLA) model driving a humanoid needs to learn from footage that looks like what its head camera will actually see: hands entering the frame from below, objects at counter height, occlusions caused by the manipulator itself. That viewpoint is called egocentric, and until recently there was no scaled, systematic way to collect it. Our thesis, argued with numbers throughout this guide: egocentric capture is the only collection method that scales to foundation-model volumes, but it earns that scaling only when modality mix, viewpoint geometry, and annotation depth are derived from the training mechanism rather than from a hardware catalog.

This guide covers the full stack: what egocentric data collection for robotics actually is, the modalities that matter (mono, stereo, depth, IMU, gaze, hand pose), the hardware options from a $500 Quest 3 to Aria Gen 2 research glasses, how the data plugs into policy training, and what it costs per hour. We include the benchmark numbers we use internally at DexSet, because pricing opacity is the single biggest complaint we hear from buyers.

DexSet supplies egocentric, exocentric, teleoperation, mono, and stereo data to physical AI teams. We have built and rebuilt the capture rigs, the QA pipelines, and the annotation stacks described below, and most of the numbers in this guide come from our own production logs.

TL;DR: Key Takeaways – Egocentric data collection captures first-person visual and sensor streams from a camera mounted at the head or chest of a human (or robot), matching the viewpoint a robot policy will see at inference time. – It is the most scalable source of manipulation pretraining data: a human wearing glasses collects demonstrations 3 to 5 times faster than a teleoperator on a bimanual rig, at roughly one third to one half the cost per hour in our benchmarks. – Hardware ranges from ~$500 (Meta Quest 3, GoPro head mounts) to research-grade Aria Gen 2 glasses with calibrated multi-camera, IMU, eye tracking, and on-device machine perception. – Egocentric human video does not replace teleoperation; the strongest results (EgoMimic, co-training pipelines behind modern VLAs) combine both. The action gap between human hands and robot grippers is the core technical problem. – Our production benchmarks: raw egocentric capture runs $15 to $22 per hour; fully annotated (hand pose, object tracks, temporal segmentation) runs $30 to $40 per hour. Teleoperation runs $28 to $60 per hour depending on rig and task complexity.

What Is Egocentric Data Collection for Robotics?

Egocentric data collection for robotics is the practice of recording synchronized video and sensor streams from a first-person viewpoint, typically a head-mounted or chest-mounted camera worn by a human demonstrator, to train robot perception and control models. The defining property is viewpoint: the camera sees the scene the way an embodied agent sees it, with the demonstrator’s own hands and workspace in frame.

Three properties separate egocentric robotics data from ordinary first-person video:

  1. Sensor completeness. A YouTube cooking clip is RGB only. A robotics-grade egocentric recording carries calibrated camera intrinsics and extrinsics, IMU streams for ego-motion, and often stereo pairs or depth so that 3D structure can be recovered.
  2. Action recoverability. The footage must support extraction of what the hands did: 3D hand pose, object 6-DoF tracks, contact events. Without recoverable actions, egocentric video is only useful for representation pretraining, not policy learning.
  3. Task intent. Recordings are organized into episodes with defined start states, goals, and outcomes, mirroring how robot demonstration datasets like those in Open X-Embodiment are structured (arXiv:2310.08864).

The reference datasets here are Meta’s Ego4D, 3,670 hours of daily-life egocentric video across 74 locations (arXiv:2110.07058), and EgoExo4D, which pairs egocentric and exocentric views of skilled activities with dense annotations (arXiv:2311.18259). Both were built for video understanding research; robotics teams now treat them as the template for what scaled first-person capture looks like.

Egocentric vs. Exocentric: Why Viewpoint Determines Value

Exocentric data is footage captured from an external, third-person viewpoint, such as a tripod camera watching a workbench, while egocentric data is captured from the agent’s own point of view. The distinction matters because a policy trained purely on third-person views must solve an extra correspondence problem at deployment: mapping an external observation of a scene onto its own body frame.

The relationship chain that matters for buyers runs like this: egocentric human video teaches visuomotor priors, teleoperation (through systems like ALOHA, arXiv:2304.13705) provides robot-embodiment action labels, imitation learning consumes both, and VLA models such as OpenVLA (arXiv:2406.09246) and π0 (arXiv:2410.24164) sit at the top of the stack. EgoExo4D demonstrated why you often want both viewpoints of the same episode: the exocentric view disambiguates whole-body motion that the egocentric camera cannot see.

In our pipelines, paired ego-exo capture adds roughly 20 to 30 percent to per-hour cost (a second calibrated camera, cross-view sync, extra QA) and is worth it for whole-body humanoid work. For tabletop manipulation, egocentric plus a single fixed reference camera is usually sufficient.

Core Modalities in Egocentric Capture

A modality is one synchronized sensor stream within a recording, and the modality mix determines both the cost of capture and what training objectives the data can support. The five that come up in nearly every RFP we see:

Mono RGB. A single color stream is the cheapest to capture and the only modality most internet-scale pretraining uses. Sufficient for representation learning and video prediction, insufficient on its own for metric 3D.

Stereo RGB. Two horizontally offset cameras allow metric depth recovery through disparity. Stereo is the workhorse for manipulation because grasp points need metric accuracy. Devices like the Intel RealSense D435i and D455 provide hardware-synced stereo pairs plus an onboard IMU; the D455’s wider baseline (95 mm vs. 50 mm) improves depth accuracy at counter-to-room distances.

Depth. Active or computed depth gives per-pixel range directly. Active IR depth degrades in sunlight and on reflective surfaces, which is why most of our outdoor captures rely on passive stereo instead.

IMU. Accelerometer and gyroscope streams recover head motion, enabling ego-motion compensation and SLAM. Project Aria glasses carry two IMUs precisely because ego-motion is that important for downstream 3D reconstruction (projectaria.com).

Gaze and hand pose. Eye tracking (available on Aria) reveals attention targets before the hand moves, and 3D hand pose is the raw material for retargeting human demonstrations to robot grippers. These are the modalities that convert “video” into “demonstration.”

Hardware: The 2026 Egocentric Rig Landscape

An egocentric capture rig is the wearable hardware package (cameras, IMU, compute, mounting) used to record first-person data, and rig choice is the largest single driver of both data quality and program cost. The four setups we run or evaluate most often:

Rig Approx. Hardware Cost Sensors Video Spec (Typical Capture Config) Calibration Best For
Meta Quest 3 (Passthrough Capture) ~$500 Stereo RGB passthrough, IMU, hand tracking 1280×1280 per eye class, 30 fps effective capture Factory, limited access Budget hand-tracked demos, teleop UI doubling as capture
Aria Gen 2 Research Glasses Research program device (not retail) RGB + mono SLAM cameras, 2 IMUs, eye tracking, spatial mics, on-device hand tracking RGB up to 8 MP class, SLAM cams at high frame rate Full factory calibration + MPS services Research-grade egocentric corpora, gaze + hand pose at scale
GoPro Head/Chest Mount $350–$550 Mono RGB (wide FOV), IMU Up to 5.3K, we typically run 4K/60 or 2.7K/120 Self-calibrated (checkerboard) High-volume, low-cost mono capture; harsh environments
RealSense D435i/D455 Helmet Rig (Custom) $700–$1,200 built Stereo IR + RGB, active depth, IMU 848×480 depth at 90 fps or 1280×720 at 30 fps, RGB 1080p Manual, per-rig Metric depth for manipulation, sim-to-real alignment

Three field notes from running these at volume:

  • Quest 3 is underrated as a capture device because its hand tracking gives you approximate 3D hand pose for free, but passthrough capture access is constrained and image quality trails dedicated cameras.
  • Aria Gen 2 is the quality ceiling. Factory-calibrated multi-camera plus eye tracking plus machine perception services means far less post-processing on our side. Access runs through Meta’s research program rather than retail channels, which affects fleet scaling plans.
  • GoPro rigs win on ruggedness and unit economics. The cost is downstream: no depth, so you pay in annotation and 3D lifting compute instead of hardware.

How Egocentric Data Trains Robot Policies

Egocentric data enters robot learning through three mechanisms: representation pretraining, action retargeting, and co-training with robot demonstrations. Understanding which mechanism you are buying data for should drive every spec decision.

Representation pretraining. Visual encoders pretrained on large egocentric corpora like Ego4D transfer to manipulation tasks better than encoders trained on third-person or object-centric images, because the visual statistics (hands, near-field objects, ego-motion blur) match deployment. This is the lowest-risk use of egocentric data: mono RGB is enough, and annotation requirements are light.

Action retargeting. Human hand trajectories extracted from egocentric video are mapped onto robot end-effectors, turning passive video into pseudo-demonstrations. This requires recoverable 3D hand pose, which is why gaze-and-hand-instrumented devices matter. EgoMimic (arXiv:2410.24221) showed that egocentric human data captured on Aria glasses, combined with robot data, improves manipulation policies over robot data alone.

Co-training. Modern VLA training mixes robot episodes (teleop, in formats like the LeRobot dataset standard, github.com/huggingface/lerobot) with human egocentric episodes in one curriculum. The human data supplies breadth of scenes and objects; the robot data anchors the action distribution to the target embodiment. Cross-embodiment training in Open X-Embodiment established the pattern that heterogeneous data mixtures beat single-source datasets, and egocentric human video is the cheapest heterogeneity you can add.

The failure mode to respect: the embodiment gap. Human wrists have degrees of freedom robot grippers lack, human reach and eye height differ from most robot platforms, and human demonstrators exploit compliance no rigid arm has. Data collection protocols can shrink this gap (constrained grasps, robot-plausible motion instructions, matched camera height), and we bake those constraints into our capture scripts.

Collection Approaches Compared: In-House, Crowdsourced, Vendor

A collection approach is the operational model used to produce the data: who wears the rig, who designs the tasks, and who owns QA. Most teams land on one of three models, and the trade-offs are stable across every program we have run.

Approach Cost per Finished Hour (Our Benchmarks) Throughput Ramp Quality Control Where It Breaks
In-house Capture Team Typically well above vendor rates once salaries, rig fleet, and management overhead are loaded in; often roughly double Slow: 2–3 months to steady state Tight, iterative Scaling past ~10 collectors; hiring drag
Crowdsourced / Distributed Low headline rate, before rejection Fast but noisy Weak; rejection rates at or beyond the top of our 10 to 30 percent planning band are common in our audits Calibration, sync, task compliance
Specialist Vendor (DexSet Model) $15–$40 fully QA’d, annotation-dependent 2–4 weeks to first delivery Contractual, sampled + automated Task designs needing daily iteration with your researchers

The honest read: in-house wins when your task distribution changes weekly and researchers need to redesign protocols on the fly. A vendor wins when the task list is stable and the bottleneck is volume with consistent QA. Crowdsourcing looks cheap until you price the rejection rate and the engineering time spent triaging unsynced, uncalibrated footage.

Cost and Economics: What Egocentric Data Actually Costs

The cost of egocentric data is best expressed as dollars per finished, QA-passed hour at a defined annotation depth, because raw capture is a minority of total program cost. Competitors rarely publish numbers, so here are ours. These are current DexSet benchmark ranges, stated as typical figures we see across programs, not quotes:

Line Item Typical Range (Per Finished Hour) Notes
Raw Egocentric Capture (mono/stereo, IMU, episode structure) $15–$22 Collector time, rig amortization, upload, storage
+ Temporal Annotation (task/step segmentation, outcome labels) +$5–$8 Largely tooling-assisted
+ 3D Hand Pose + Object Tracks +$8–$12 The expensive layer; drives the $30–$40 fully-annotated figure
Paired Ego + Exo Capture +20–30% on capture line Second camera, cross-view sync, extra QA
Teleoperation (for comparison) $28–$60 Rig and task complexity dependent; bimanual fine manipulation sits at the top

Two planning rules of thumb from our production logs:

  • Budget 15 to 25 percent of hours for QA failure. Motion blur, dropped IMU packets, and off-task episodes are facts of life. Vendors should absorb this; if you collect in-house, plan for it.
  • Annotation depth should follow the training mechanism. If you are pretraining encoders, do not pay for hand pose. If you are retargeting actions, hand pose is the whole point. We regularly see RFPs over-specified by $10+ per hour because annotation depth was copied from a paper rather than derived from the training plan.

One more line item buyers forget: storage and delivery. Stereo capture at 30 fps with IMU sidecars generates roughly 50 to 120 GB per hour depending on resolution and compression, so a 5,000-hour corpus is a few hundred terabytes before derivatives. Cloud egress on a corpus that size is real money, which is why we quote delivery format and transfer method inside the per-hour price rather than as a surprise on the final invoice. Ask any vendor to do the same.

At these rates, a 5,000-hour egocentric corpus with full annotation lands between $150K and $200K. The equivalent volume via bimanual teleoperation would run $140K to $300K and take 3 to 5 times as long on a comparable rig fleet, which is the arithmetic behind the current industry shift toward human egocentric pretraining with a smaller teleop fine-tuning set.

Case Study Proof: 4,000 Hours for a Humanoid VLA Team

A humanoid foundation model team came to us with a stalled co-training experiment: their teleop corpus was high quality but topped out near 400 hours, and scaling it 10x on their own rigs would have taken most of a year. We scoped a 4,000-hour egocentric program across kitchen, warehouse shelving, and assembly-bench task families, captured on stereo rigs at camera heights matched to their robot’s head frame, with hand pose and object tracks on the 30 percent of hours their researchers flagged as retarget-critical.

Delivery ran 14 weeks. Their team reported that co-training on the mixed corpus improved task success on unseen-object manipulation evaluations relative to their robot-only baseline, consistent with the direction published in EgoMimic-style co-training work. The full breakdown of task families, QA gates, and the capture protocol is in the case study blog this pillar links to below.

Related reading from this series: – Why Egocentric Data Collection for Robotics Is the Biggest Bottleneck in Physical AI – Comparing Egocentric Data Collection Approaches: Pros, Cons and Costs – Case Study: Scaling Egocentric Data Collection for a VLA Model – 5 Hidden Challenges in Egocentric Data Collection

Download: The Egocentric Data RFP Template

An RFP template turns this guide into a procurement tool: it lists the 40 questions we believe every buyer should ask a data vendor, covering rig specs, calibration evidence, sync tolerances, QA sampling methodology, annotation rubrics, pricing structure, and data licensing. We built it from the RFPs we answer, including the questions we wish more buyers asked. Download it, delete our name from the header if you like, and send it to every vendor on your shortlist including us.

Put These Numbers to Work

If you are scoping an egocentric data program this quarter, two options. Download the RFP template and pressure-test every vendor with it, or book a 30-minute demo and we will walk you through sample episodes from our stereo and Aria-class rigs, including the QA reports we ship with every batch. Either way, you leave with real numbers instead of a sales deck.
Request a Quote

Frequently Asked Questions

What is egocentric data collection for robotics?

Egocentric data collection for robotics is the recording of synchronized first-person video and sensor streams (RGB, stereo, depth, IMU, gaze, hand pose) from head- or chest-mounted rigs, structured into task episodes, to train robot perception and manipulation models.

Egocentric data captures a human performing tasks with their own hands from a first-person camera, while teleoperation data captures a robot performing tasks under human control, with exact joint-space action labels. Egocentric data is cheaper and faster to scale; teleoperation data matches the robot embodiment exactly. Most modern VLA pipelines use both.

Common rigs include Meta Quest 3 (~$500, stereo passthrough and hand tracking), Aria Gen 2 research glasses (calibrated multi-camera, IMU, eye tracking), GoPro head or chest mounts ($350 to $550, mono RGB), and custom helmet rigs built around Intel RealSense D435i or D455 stereo depth cameras.

In DexSet’s benchmarks, raw QA-passed egocentric capture runs $15 to $22 per hour, and fully annotated data with hand pose and object tracks runs $30 to $40 per hour. Teleoperation data runs $28 to $60 per hour for comparison.

No. Egocentric human video scales pretraining and improves generalization, but the embodiment gap between human hands and robot grippers means policies still need robot-embodiment data (teleoperation or autonomous rollouts) for reliable control. Research such as EgoMimic supports combining both.

It depends on the mechanism: encoder pretraining benefits from thousands of hours of lightly annotated video, while retargeting pipelines often start showing gains with hundreds of hours of densely annotated, task-matched capture combined with a robot demonstration set.