In the kitchen bay of our capture floor, a collector claps once in front of her helmet camera, waits for the sync check to pass on the wall monitor, and begins unloading a dishwasher using only grasps a parallel-jaw gripper could copy. By the end of her shift she will have produced more usable first-person manipulation data than one of our bimanual teleoperation stations yields in a week. Nothing about that scene is exotic. The exotic thing is how rare it still is.
It is rare for a structural reason. Text and images were exhaust from human activity; someone else already paid to produce them. First-person manipulation data is not exhaust. Nobody films their hands doing the dishes with calibrated stereo cameras and synchronized IMUs, so every hour has to be manufactured deliberately, by someone wearing a rig or driving a teleoperation station, and then inspected before it is fit to train on. The largest open robot manipulation dataset, Open X-Embodiment, took 21 institutions pooling 22 embodiments to reach roughly a million trajectories (arXiv:2310.08864), while language models trained on trillions of tokens.
This post argues one thesis: the binding constraint on physical AI right now is the supply of QA-passed egocentric hours, not compute and not architecture. You will get the arithmetic behind that claim, our cost benchmarks for the three ways teams try to break the constraint, and a checklist for scaling collection without torching your budget.
We spend our days at DexSet manufacturing exactly this data: egocentric, exocentric, and teleoperation streams for VLA and humanoid teams. The numbers below come from our production logs, not analyst reports.
Key Takeaways
- Robot learning is data-bound: internet-scale text corpora dwarf the largest robot manipulation datasets by many orders of magnitude, and almost no internet video is first-person with recoverable actions.
- Teleoperation alone cannot close the gap. A well-run bimanual station yields roughly 15 to 25 usable episode-hours per week; at $28 to $60 per hour, 10,000 hours is a multi-year, seven-figure program.
- Egocentric human capture shifts the curve: $15 to $40 per hour in our benchmarks, 3 to 5x the collection rate per operator, and it scales with people instead of robot fleets.
- The constraint is not raw hours, it is QA-passed hours. Plan for 15 to 25 percent of captured footage to fail quality gates.
- Scaling works when you treat collection as a manufacturing process: fixed protocols, instrumented rigs, sampled QA, and annotation depth matched to the training objective.
Why Data, Not Compute, Is the Binding Constraint
The binding constraint in a training pipeline is the input you cannot buy more of at market price, and for physical AI that input is embodiment-relevant data. Compute is rentable by the hour. Architectures are published within months of working. But there is no marketplace where you can order 10,000 hours of first-person manipulation with hand pose annotations for next quarter, because until recently the industry that manufactures it did not exist.
Look at what the flagship models actually consumed. RT-2 leaned on internet-scale vision-language pretraining precisely because robot data was scarce (arXiv:2307.15818). OpenVLA trained on 970K robot episodes from Open X-Embodiment (arXiv:2406.09246). π0 combined internet-scale pretraining with a proprietary dexterous manipulation corpus (arXiv:2410.24164). The pattern across all three: model capability tracked data supply, and every team supplemented public data with expensive private collection.
Meanwhile the largest first-person video corpus in existence, Ego4D, holds 3,670 hours (arXiv:2110.07058). Impressive for a research effort, and still less footage than a single mid-size YouTube channel uploads in a year. For robotics, most of it lacks the calibration and action annotations that policy learning needs. The shelf is nearly empty. Whoever wants the data has to make it.
The Teleoperation Ceiling
The teleoperation ceiling is the practical throughput limit of collecting robot demonstrations through human-controlled rigs, and it is lower than most roadmaps assume. ALOHA-style bimanual stations (arXiv:2304.13705) made teleop dramatically cheaper to build; the ALOHA rig famously targeted a ~$20K bill of materials against six-figure predecessors. What they did not change is the physics of throughput.
In our teleop operations, a trained operator on a healthy rig produces 15 to 25 usable episode-hours per week once you subtract resets, calibration, failed episodes, and operator fatigue. Push operators harder and quality drops before throughput rises; fine manipulation under fatigue produces exactly the jerky, hesitant trajectories you do not want a policy imitating.
Now run the arithmetic for a 10,000-hour corpus, which is a modest target by foundation model standards:
| Approach | Cost per QA-passed hour (our benchmarks) | Realistic throughput | Time to 10,000 hours | Program cost |
|---|---|---|---|---|
| Teleoperation, 5 bimanual rigs | $28 to $60 | 75 to 125 hrs/week | 1.5 to 2.5 years | $280K to $600K |
| Teleoperation, 20 rigs | $28 to $60 | 300 to 500 hrs/week | 5 to 8 months | $280K to $600K + fleet capex |
| Egocentric human capture, 20 collectors | $15 to $40 | 400 to 700 hrs/week | 4 to 6 months | $150K to $400K |
The 20-rig teleop row hides the real problem: rig fleets need robot hardware, lab space, maintenance engineers, and operator pipelines. Twenty humans wearing $500 to $1,200 capture rigs need a protocol document and a QA queue. That asymmetry is the entire argument for egocentric capture as the scaling layer, with teleop reserved for the embodiment-matched fine-tuning set. Systems like EgoMimic have shown the combined recipe outperforms robot data alone (arXiv:2410.24221).
Why Egocentric Capture Is Hard to Scale Anyway
Scaling egocentric collection means turning a research-style capture activity into a manufacturing process with yield metrics, and most attempts stall on the same four problems.
Yield, not volume. Raw hours are easy; QA-passed hours are the product. Across our programs, 15 to 25 percent of captured footage fails gates for motion blur, dropped IMU packets, sync drift, or off-protocol behavior. Teams that do not measure yield discover it at training time, which is the most expensive place to discover it.
Calibration debt. Every uncalibrated GoPro hour you collect today is an hour of 3D lifting problems you inherit later. Devices like Aria carry factory calibration and machine perception services (projectaria.com); consumer rigs need per-unit calibration routines and periodic re-checks, and skipping them is the most common false economy we see.
Annotation as the cost center. Capture at $15 to $22 per hour feels cheap until hand pose and object tracks push finished cost to $30 to $40. The fix is not cheaper annotation; it is refusing to annotate hours your training plan does not need at that depth.
Protocol drift. Thirty collectors interpret “tidy the workspace” thirty ways. Without scripted task cards, fixed start states, and outcome labels, you end up with footage instead of episodes. The LeRobot dataset conventions (github.com/huggingface/lerobot) are a useful forcing function here: if a capture cannot be expressed as episodes with consistent schema, it probably was not collected with enough discipline.
A Scaling Checklist That Survives Contact With Reality
A scaling checklist is only useful if each item maps to a failure we have actually watched happen, so here is ours:
- Define the training mechanism first (encoder pretraining, retargeting, co-training), then derive modalities and annotation depth from it.
- Set a QA yield target (we hold vendors, including ourselves, to 80 percent+ first-pass) and sample-inspect every batch.
- Match camera height and field of view to the deployment embodiment before collecting hour one.
- Script tasks as episodes: start state, goal, outcome label, retry policy.
- Calibrate on a schedule, not on failure.
- Track cost per QA-passed hour weekly; it is the only number that predicts your budget.
- Pilot 50 to 100 hours and train on them before committing to thousands.
If you want the full treatment of rigs, modalities, and economics behind this checklist, it lives in our pillar guide: The Complete Guide to Egocentric Data Collection for Robotics.
Frequently Asked Questions
Why is data the bottleneck in physical AI rather than compute?
Compute and architectures are purchasable and published, while first-person manipulation data with recoverable actions barely exists on the internet and must be manufactured deliberately. The largest robot datasets hold on the order of a million trajectories, orders of magnitude below the corpora that trained large language models.
How fast can teleoperation generate training data?
In DexSet’s operations, a trained operator on a bimanual rig produces roughly 15 to 25 usable episode-hours per week after resets, failures, and fatigue. Scaling further requires more robot rigs, lab space, and maintenance, which is why teleop fleets scale slowly.
How much cheaper is egocentric capture than teleoperation?
Our benchmarks put QA-passed egocentric capture at $15 to $40 per hour depending on annotation depth, against $28 to $60 per hour for teleoperation, with 3 to 5 times higher collection throughput per operator.
Does egocentric human data remove the need for robot data?
No. The embodiment gap between human hands and robot grippers means policies still need robot-embodiment demonstrations. The strongest published results co-train on both, as in EgoMimic.
What QA failure rate should I budget for?
Plan for 15 to 25 percent of raw captured footage to fail quality gates for blur, sync drift, sensor dropouts, or protocol violations.
From Bottleneck to Plan
If your roadmap has a data line item and a question mark next to it, start with the pillar guide above, then request a sample batch from us. We ship real episodes with real QA reports, and the pricing table is on the first page, not behind a discovery call.
