Skip to main content

Dexset

dexset
Menu
Request a Demo

Tag: VLA Models

Case Study: Scaling a VLA Training Data Program from 500 to 12,000 Usable Hours Without Doubling the Budget

Posted on by Enoch Pakanati

Cell F14 of the spreadsheet a VLA startup sent us in January read cost_per_usable_hr = $71. Two rows down sat the scaling plan: 12,000 usable hours by July, up from the 500 they had collected on an internal ALOHA-class cell before their policy plateaued. At $71 per usable hour, the plan cost $852,000. The approved budget in the adjacent column was $560,000. The engagement began, literally, as a request to make F14 say something else.

The gap behind that cell was not caused by anything exotic. Their single rig sat idle 60 percent of the day. Their rejection rate was 31 percent because task specs lived in a Slack thread. And every hour they collected was premium bimanual teleop, even the hours that existed only to teach the visual encoder what countertops look like.

This is the most common shape of scaling problem we see, and it carries the thesis this case study argues: data budgets are rescued by operational variables, rejection rate, modality mix, and rig utilization, not by rate negotiation. Names and identifying details are removed at the client’s request; every number is from the actual program ledger.

What follows: the starting cost baseline, the three changes that moved it, the month-by-month rejection curve, and the final economics. If you are scaling any demonstration data program past a few thousand hours, at least one of these three changes applies to you.

Key Takeaways

  • Starting point: $71 per usable hour, 31% QA rejection, one under-utilized rig. Ending point: $46 per usable hour, 11% rejection, 12,100 usable hours delivered.
  • Fix 1: a written, versioned task spec cut rejection from 31% to 14% in five weeks. Cost: about a week of engineering time.
  • Fix 2: a 65/35 modality split (egocentric pretraining at $22/hr blended, teleop post-training at $44/hr blended) cut the blended rate by 29%.
  • Fix 3: two-shift rig utilization dropped amortization per hour from $11 to $5.
  • Total spend: $557,000 against a $560,000 budget. The policy’s task success rate improved 19 points on their internal eval.

The Baseline: Why $71 per Usable Hour Happens

A cost baseline is the fully loaded spend per QA-passed hour at the start of an engagement, and it is almost always worse than the team believes because rejection losses hide inside it. The startup quoted their internal cost as “about $49 an hour.” That was spend divided by raw hours. Divided by usable hours, it was $71.

Their baseline decomposed like this:

Line item Their baseline Notes
Rig amortization $11/raw hr One ~$26k cell, single shift, idle 60% of the day
Operator labor $29/raw hr Two part-time operators, still early on the learning curve
QA review $9/raw hr Manual review, no automated pre-checks
Raw total $49/raw hr The number they quoted internally
Rejection rate 31% Ambiguous specs, no calibration checklist
Usable total $71/usable hr $49 / (1 - 0.31)

Nothing in that table is negligence. It is what a research-grade pipeline looks like when a training run suddenly demands production volume. The ALOHA hardware itself was fine; a roughly $20k bimanual rig is a proven design (https://arxiv.org/abs/2304.13705). The economics around the hardware were the problem.

Fix 1: A Versioned Task Spec Cut Rejection by More Than Half

A task specification is a written, versioned document defining episode start states, success criteria, camera framing requirements, and reset procedures, and it is the highest-ROI artifact in any capture program. Theirs was a Slack thread and tribal memory. Operators guessed, and QA rejected the guesses.

We replaced it with a two-page spec per task family: exact object sets, start-state photos, an explicit success predicate, and a pre-episode calibration checklist. Rejections got tagged with a reason code so the spec could be patched weekly.

The rejection curve over the program:

Month Rejection rate Main rejection driver
Jan (baseline) 31% Ambiguous success criteria
Feb 22% Camera framing drift
Mar 14% Occluded end-effector
Apr 12% Long-horizon task failures
May to Jul 11% Steady state

Total cost of the fix: roughly one engineer-week plus 30 minutes of weekly maintenance. Effect: every subsequent dollar bought 29 percent more usable data. No other intervention in this program came close on ROI.

Fix 2: Stop Buying Premium Hours for Non-Premium Jobs

Modality staging is the practice of assigning cheap capture modalities to pretraining objectives and reserving teleoperation for action-supervised post-training. The startup was training their visual encoder on $49-per-hour bimanual teleop footage. The encoder did not care that the footage had action labels; it needed diverse pixels of hands, objects, and scenes.

We restructured the volume: 65 percent egocentric human video for representation pretraining, collected across 40+ real kitchens and workspaces at a blended $22 per hour, and 35 percent bimanual teleop for post-training at a blended $44 per hour. This mirrors what the public record already suggested: cross-source mixtures train stronger generalists, which is the entire premise of Open X-Embodiment’s 1M+ trajectories across 22 embodiments (https://arxiv.org/abs/2310.08864) and DROID’s 76k episodes across 52 buildings (https://arxiv.org/abs/2403.12945).

Blended raw rate after the split: $29.70 per hour, a 29 percent reduction against their all-teleop plan. Their ML lead’s initial worry, that ego data would not move policy metrics, resolved in evaluation: the encoder pretrained on the ego corpus reached the same downstream success with 22 percent fewer teleop post-training hours. The cheap data made the expensive data go further.

Fix 3: Utilization Is a Pricing Decision

Rig utilization is the fraction of available hours a capture station spends collecting, and it directly divides your amortization cost per hour. One cell on one shift with 60 percent idle time was charging them $11 per hour just for hardware.

Changes: we added a second shift on their existing cell, stood up two DexSet production cells in parallel, and scheduled resets and calibration into shift handovers instead of collection time. Amortization dropped to $5 per hour. Operator throughput also compounded, as it always does; our operators improve 30 to 50 percent over their first 200 hours, and by April the program’s episodes-per-shift was up 41 percent against baseline.

The Final Ledger

Program economics are only credible as a closed ledger, so here is the whole thing:

Item Plan (their original) Actual (restructured)
Usable hours target 12,000 12,100 delivered
Blended raw rate $49/hr (all teleop) $29.70/hr (65/35 mix)
Rejection rate 31% 11% steady state
Cost per usable hour $71 $46
Language annotation not budgeted $10/usable hr on teleop subset ($42,400)
Total spend $852,000 projected $557,000 actual

Against their internal eval suite, task success improved 19 points over the plateaued January checkpoint. We do not claim the data mix deserves all of that credit; their team shipped model improvements in the same window. We do claim the counterfactual: at $71 per usable hour, the volume that enabled those experiments would not have fit the budget.

Every formula behind this ledger, including the cost-per-usable-hour tables and the rig amortization model, is published in our robot training data costs and pricing guide.

What Transfers to Your Program

A transferable lesson is one that holds regardless of embodiment or task list, and this program had three. Write the task spec before you scale, because rejection rate is the multiplier on everything. Split modalities by training objective, because encoders do not need action labels. And treat utilization as pricing, because an idle rig is a fee you pay for nothing. None of these require our involvement to implement. They do require someone to own cost per usable hour as a weekly metric.

Apply This Playbook to Your Program

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

How many usable hours does a VLA program actually need?

It varies by task breadth and model scale, but the programs we run for post-training typically land between 5,000 and 30,000 usable teleop hours, on top of larger and cheaper pretraining corpora. Budget from your eval targets backward, not from a round number.

With a versioned task spec and calibration checklists, 10 to 15 percent is a realistic steady state in our pipelines. Above 25 percent, fix your spec before you buy another hour.

In this program, encoder pretraining on egocentric video reached equal downstream success with 22 percent fewer teleop post-training hours. Results vary with task overlap, but the direction is consistent across our engagements.

$46, down from a $71 baseline: a 35 percent reduction from spec discipline, modality staging, and rig utilization combined, with no reduction in QA standards.

The task spec and utilization fixes, yes, immediately. The modality split needs access to egocentric collection at scale, which is harder to stand up internally; that is typically where teams engage a provider.

The Robot Training Data Costs and Pricing Guide contains every benchmark and formula used in this program. Or book a scoping call and we will baseline your current cost per usable hour in one session.

Why Data Procurement, Not Data Capture, Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

Two teams approached us in the same quarter with nearly identical VLA data programs. The first sent a two-page acceptance spec and an RFP that demanded numbers, then ran a paid pilot before signing with anyone. The second compared three price sheets and took the lowest. Half a year later, the first team’s deliveries were entering training runs the day they arrived; the second team was still bisecting a training regression that traced back to a sync tolerance no contract had ever specified. Same budget class, same architecture, divergent quarters. The difference was not capture quality. It was procurement.

The uncomfortable part is that capture itself has scaled. ALOHA-class rigs are reproducible from public documentation (arxiv.org/abs/2304.13705). Open X-Embodiment pooled over a million episodes across 22 embodiments (arxiv.org/abs/2310.08864). DROID collected 76,000 teleop episodes across 13 institutions (arxiv.org/abs/2403.12945). The hardware and process knowledge exist. What has not scaled is the buying side: most teams still purchase demonstration data with less rigor than they apply to a laptop refresh.

This post makes the case that procurement is now the binding constraint, shows what the gap costs in numbers, and gives you the four artifacts that close it. It condenses the full Robotics Data Buyer’s Playbook, which includes the complete scorecard and RFP question bank.

We see this from the vendor side at DexSet. The buyers who send a real spec get better data at better prices than the buyers who send a budget and a hope, because a real spec lets us commit to numbers instead of hedging against unknowns.

Key Takeaways

  • Capture capacity has commoditized; vendor selection has not. Weak procurement is now the most common cause of stalled VLA data programs.
  • The cost of a bad pick is measured in usable hours: a cheap vendor at 70 percent yield can cost more per usable hour than a pricier one at 90 percent, once rework is priced in.
  • Latent defects (sync error, calibration drift) are invisible in previews and only surface in training, which is why sample-based buying fails.
  • The fix is procedural, not heroic: a spec-first RFP, a 10-criterion weighted scorecard, a red flags list, and a 50-hour paid pilot.

What Makes Procurement the Bottleneck

The procurement bottleneck is the delay and waste created when robot data purchasing decisions are made without a specification, a scoring method, or a pilot, forcing quality problems to surface downstream during training. It shows up as three concrete failure patterns.

Pattern one: the invisible defect. Robot data defects are latent. A 40 ms sync offset between camera frames and joint states will not appear in video playback, but it corrupts the state-action mapping your imitation learning policy depends on. Teams that buy on sample previews systematically miss this class of problem, then spend weeks bisecting training regressions that were purchased, not coded.

Pattern two: the spec vacuum. When the buyer has no written acceptance spec, every vendor quote describes a different product. One vendor’s “hour of manipulation data” is stereo egocentric plus wrist cameras with dense annotations; another’s is a single 720p exocentric mono stream. Comparing their prices is meaningless, and procurement stalls in clarification loops that a two-page spec would have prevented.

Pattern three: the format tax. Deliveries arrive in whatever the vendor uses internally: undocumented HDF5, half-converted rosbag2, a proprietary container. Your engineers pay the conversion cost into LeRobot or RLDS on every delivery. In our experience that tax runs 15 to 25 percent of program cost when format compatibility was never contracted.

The Cost of Buying Badly, in Numbers

The cost of weak procurement is best expressed as price per usable hour, which is the delivered price divided by the fraction of hours that pass your acceptance spec. The sticker price is the number vendors compete on; the usable-hour price is the number your training run experiences.

Scenario Sticker price Usable-hour yield True price per usable hour 1,000-hour program cost (usable basis)
Cheapest bid, no pilot $32/hr 70% $45.71 $45,710
Mid bid, sample-only check $40/hr 82% $48.78 $48,780
Higher bid, passed 50-hr pilot $46/hr 90% $51.11 $51,110
Cheapest bid after rework and triage time $32/hr + engineering time 70% $55–$60 effective $55,000–$60,000

Read the last row carefully. The cheapest vendor is the most expensive one once you price the engineering time spent triaging failures and patching training mixes, and that is before counting the schedule slip, which no spreadsheet captures but every roadmap feels. These are typical ranges from our benchmarks; your yields will vary, which is exactly why you measure them in a pilot instead of assuming them.

The Fix: Four Artifacts, Not More Meetings

The fix for the procurement bottleneck is a set of four reusable artifacts that convert vendor selection from judgment calls into measurements. Each one is small. Together they remove the guesswork that creates the bottleneck.

  • A spec-first RFP. Write the acceptance spec (cameras, sync tolerance under 10 ms, task list, annotation rubric, delivery format) before contacting anyone. Then send 15 questions that force numeric answers: committed usable-hour yield, audit accuracy, rig counts, per-hour pricing by task complexity.
  • A weighted scorecard. Ten criteria, weights summing to 100, with data quality SLAs, modality coverage, and calibration/sync spec carrying 39 points between them. Two independent scorers, reconciled. Automatic disqualification for weak answers on quality, calibration, or licensing.
  • A red flags list. Some behaviors end the conversation regardless of score: pricing only after discovery calls, refusal of paid pilots, proprietary-only formats, no consent documentation, broad data reuse rights.
  • A 50-hour paid pilot. Production rigs, production operators, and four predefined numbers: usable-hour yield (85 percent or higher), annotation audit accuracy (97 percent or higher on a 5 percent sample), policy success delta on a fixed eval set, and loader time into your LeRobot or RLDS pipeline (one engineer-day or less).

The full versions, including the complete scorecard weights and all 15 RFP questions, are in the Robotics Data Buyer’s Playbook.

How to Tell If Procurement Is Your Bottleneck

A procurement bottleneck diagnosis is a check of where data-program time actually goes, and it takes one honest hour with your delivery logs. Run through five questions:

  • Can anyone state your acceptance spec from memory, or point to the document? If the spec lives in tribal knowledge, every vendor conversation is renegotiating it implicitly.
  • What fraction of delivered hours reached a training run last quarter? If nobody tracks this number, assume it is worse than you think; teams that start measuring usually find 15 to 30 percent of purchased hours never trained anything.
  • How long does a delivery take to enter the pipeline? More than a few hours of engineer time per batch means you are paying the format tax on every delivery.
  • Could you defend your current vendor choice to your board with numbers? A scorecard produces that defense as a byproduct. A demo-based decision cannot.
  • When did a data defect last cost you a training debugging cycle? If the answer is “this quarter,” the defect entered through procurement, not through capture.

Two or more uncomfortable answers means the bottleneck is upstream of your training code, and the fix below is cheaper than the symptom.

Scaling the Playbook Across Programs

Scaling this process means running the same artifacts on every purchase rather than reinventing evaluation per deal. The spec becomes a living document versioned alongside your model releases. The scorecard weights shift as your risks shift: teams early in data collection weight modality coverage higher; teams scaling a proven recipe weight throughput and SLAs higher. Pilot results accumulate into an internal vendor database, which is the closest thing this industry has to a track record. After three or four cycles, vendor evaluation drops from weeks of meetings to days of scoring, and that is the bottleneck removed.

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

Why is procurement a bigger bottleneck than capture in physical AI?

Because capture methods are now well documented (ALOHA, DROID, Open X-Embodiment) while buying practices are not. Latent defects like sync error survive sample review and surface in training, so unstructured purchasing converts directly into stalled training runs.

Delivered price divided by the fraction of hours passing your acceptance spec. A $32/hr vendor at 70 percent yield costs $45.71 per usable hour, which is why sticker-price comparisons mislead.

Around 50 paid hours. That volume exposes process problems (calibration drift, operator variance, QA gaps) while keeping a failed pilot cheap: typically $1,500 to $4,500 at market rates.

Native delivery in LeRobot, RLDS/TFDS, or documented HDF5, with MCAP or rosbag2 as ROS 2 options. Proprietary-only delivery is a disqualifier because it adds conversion cost to every delivery and blocks independent audits.

Why Training Data Costs Are the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

“Training data” is usually defined as something you gather. That definition hides the economic fact that decides robotics budgets: robot demonstration data cannot be gathered at all. Language models scraped trillions of tokens the internet had already produced for free. A robot demonstration has to be manufactured, one episode at a time, by a person and a machine in a room, and manufactured goods have unit costs that scraping never did.

That manufacturing has a price, and the price is the thesis of this post: the binding constraint on physical AI progress is the unit economics of demonstration data, not model architecture. Our benchmarks put teleoperation at $28 to $60 per hour, egocentric human video at $15 to $40, and multi-view exocentric capture at $20 to $50, before annotation adds another $8 to $25 per pass. Multiply any of those by the hundreds of thousands of hours that scaling curves suggest, and the number stops looking like a data budget and starts looking like a Series B.

Compute costs fall on a curve you can plan around. GPU-hours get cheaper every year; teleoperator-hours do not, because they are wages plus hardware plus QA. So while everyone argues about architectures, the teams actually shipping robot foundation models are constrained by a much less glamorous question: how many usable demonstration hours can we afford this quarter?

This post breaks down why the bottleneck is economic rather than algorithmic, what the per-hour math actually looks like, and where the cost curve is bending. It draws on DexSet’s own capture operations, so the numbers are operating benchmarks, not estimates.

Key Takeaways

  • Robot data is manufactured, not scraped. Teleop costs $28 to $60 per hour; egocentric video $15 to $40; multi-view exo $20 to $50 (DexSet benchmarks).
  • QA rejection of 10 to 30 percent inflates every quoted rate. Budget on cost per usable hour.
  • Open X-Embodiment needed 21 institutions to pool 1M+ trajectories across 22 embodiments; no single lab could afford that collection alone.
  • Rig capex is the small part: an ALOHA-class station is roughly $20k and amortizes fast. Labor and QA dominate.
  • The cost curve bends through cheaper capture devices (UMI-style grippers), human video pretraining, and better data selection, not through cheaper wages.

The Bottleneck Is Economic, Not Algorithmic

The physical AI bottleneck is the gap between the demonstration volume that current methods need and the demonstration volume that current budgets can buy. Imitation learning works. ACT on ALOHA hardware showed fine bimanual manipulation from a rig that cost roughly $20k (Zhao et al., https://arxiv.org/abs/2304.13705). VLA models generalize further as data grows. The recipe is not the mystery; funding the recipe is.

Look at what it took to build the field’s reference datasets. Open X-Embodiment pooled data from 21 institutions to reach more than 1 million trajectories across 22 robot embodiments (https://arxiv.org/abs/2310.08864). DROID took a multi-lab consortium collecting across 52 buildings for a year to produce 76,000 episodes (https://arxiv.org/abs/2403.12945). These are consortium projects because the economics forced them to be. When the leading academic labs in the world have to carpool, the per-hour cost of data is the constraint worth studying.

Contrast that with a startup’s position. A humanoid company that wants 50,000 proprietary teleop hours at a blended $42 per hour is staring at a $2.1M capture bill before annotation, before storage, and before the 10 to 30 percent QA rejection rate we measure in our own pipelines pushes the real figure higher. That is the bottleneck in one sentence: the marginal trajectory costs real money, and scaling laws demand a lot of margins.

Where the Money Actually Goes

A fully loaded data cost is the sum of hardware amortization, collection labor, QA review, annotation, and infrastructure, and its composition explains why the bottleneck resists quick fixes. Hardware is the layer everyone obsesses over and the one that matters least.

Cost layer Teleoperation Egocentric video Share of total (typical)
Hardware amortization $4 to $9/hr $1 to $4/hr 10 to 15%
Collection labor $18 to $38/hr $10 to $26/hr 55 to 65%
QA and recollection $6 to $12/hr $5 to $10/hr 20 to 30%
Total (raw hour) $28 to $60/hr $15 to $40/hr 100%

Two things jump out of that table. Labor dominates, and labor does not follow Moore’s law. A teleoperator in year three costs what a teleoperator cost in year one, adjusted upward for wages. The only labor lever is throughput: in our programs, operator productivity improves 30 to 50 percent over their first 200 hours, which is real but bounded.

The second thing: QA is a fifth to a third of the bill, and it is the layer buyers most often forget. An episode fails for dropped frames, a desynced camera, an occluded gripper, or a task that did not actually complete. At a 25 percent rejection rate, a $40 quote is really $53.33 per usable hour. We walk through that math, with tables, in our full robot training data costs and pricing guide.

Why Compute Got Cheap and Data Did Not

Compute costs fall because silicon improves and utilization tooling matures, while demonstration data costs stay flat because their main input is human time in physical space. This asymmetry is the strategic fact of the next five years of robotics.

A training run you could not afford in 2023 is routine in 2026. But the demonstration hour you collected in 2023 cost about what it costs today, and the scene setup, the resets between episodes, and the review pass all still happen at human speed. Physics does not batch. You cannot checkpoint a kitchen.

The practical consequence: data spend is becoming the durable moat while compute spend becomes a commodity line item. Teams that treat their data budget with the same rigor as their compute budget, tracking cost per usable hour, rejection rates, and hours-to-policy-improvement, compound an advantage that a bigger cluster cannot erase.

Where the Cost Curve Actually Bends

Cost-curve bending in robot data comes from cheaper capture devices, cheaper modalities for pretraining, and better data selection, not from paying people less. Three developments are doing real work right now.

  • Handheld capture devices. UMI-style grippers (Chi et al., https://arxiv.org/abs/2402.10329) put a wrist camera on a portable gripper, so collection happens in real homes without a robot present. Our build estimate is under $1,000 per unit. Labor cost drops toward egocentric rates while output stays gripper-centric.
  • Human video for pretraining. Egocentric data at $15 to $40 per hour can carry representation learning, reserving expensive teleop for post-training. A 70/30 ego-to-teleop mix can cut blended cost per hour by a third without giving up action supervision where it counts.
  • Data selection over data volume. Deduplication, difficulty-aware sampling, and rejecting low-information episodes before annotation mean you pay $8 to $25 per hour of labels only on data that earns it.

None of these eliminate the bottleneck. They move the ratio of usable hours per dollar, which is the correct objective.

What This Means for Your Budget

A defensible data plan starts from cost per usable hour and works backward to volume, rather than starting from a raw-hour quote and hoping. If you take one action from this post, make it this checklist:

  • Get every vendor quote itemized across hardware, labor, QA, annotation, and infrastructure.
  • Demand a measured QA rejection rate from a comparable program, and pilot before committing volume.
  • Split your pipeline: cheap modalities for pretraining volume, teleop for action-supervised post-training.
  • Track cost per usable hour monthly. It is your burn rate’s most honest line.

Put the Numbers to Work

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

Why is robot training data more expensive than language data?

Language data was scraped from the internet at near-zero marginal cost. Robot data is manufactured: a person, a rig, and a physical scene produce one episode at a time, at $15 to $60 per hour depending on modality, plus QA and annotation.

Collection labor, at 55 to 65 percent of the fully loaded hourly rate in our programs. Hardware amortization is only 10 to 15 percent, which is why cheaper rigs alone do not fix the bottleneck.

As a reference point, 50,000 teleop hours at a blended $42 per hour is $2.1M before annotation and before QA rejection losses of 10 to 30 percent. Consortium datasets like Open X-Embodiment exist precisely because no single lab wanted to carry that cost.

It reduces it for some skills, but sim-to-real transfer still needs real-world demonstrations for contact-rich manipulation, and mixed pipelines still budget significant real capture. Treat sim as a multiplier on real data, not a replacement.

Tighten task specs to cut QA rejections, mix cheaper egocentric or UMI-style capture into pretraining, annotate only selected data, and measure rejection rates continuously. The full cost model is in our pricing guide.

The Robot Training Data Costs and Pricing Guide publishes our complete per-hour benchmarks, rig capex table, and a downloadable budget spreadsheet. No sales call required to see the numbers.

Why Exocentric & Multi-View Data Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

Language models got their training data for free: by the time the first large transformer trained, the internet had already spent decades producing trillions of tokens describing every topic from millions of viewpoints. Physical AI inherits no such gift. Robot data must be manufactured episode by episode, and most of it has been manufactured the cheap way: one camera, no calibration record, no synchronization guarantees. Compute got cheaper, architectures converged, and the gap between what models could absorb and what capture pipelines produce kept widening.

The pattern shows up concretely in teams we work with. One VLA group spent six weeks tuning architectures against a plateau. Bigger backbone, longer action chunks, better augmentation. The success rate on cluttered scenes moved two points. Then they looked at their data and found the real ceiling: every one of their 90,000 episodes was recorded from a single camera, and 40 percent of failures happened in the exact frames where that camera could not see the target object.

This post makes the case that calibrated exocentric and multi-view capture, not model design, is the binding constraint on manipulation performance right now, and shows what closing the gap costs. The evidence comes from public datasets (DROID, Open X-Embodiment, Ego-Exo4D), the RoboMimic observation-space study, and our own capture benchmarks at DexSet, where multi-view teleoperation rigs are what we run every day.

Key Takeaways – Single-view episodes cap policy performance on occlusion-heavy tasks regardless of architecture; the failure is in the data, not the model. – DROID made two external stereo views plus wrist a hard protocol requirement across 76,000 episodes; Open X-Embodiment’s viewpoint chaos shows what happens without such a standard. – The bottleneck is operational, not scientific: calibration drift, sync skew, and 3-4x storage are why teams default to one camera. – Our benchmarks: DROID-style rigs cost $4,500 to $7,000 to build and $26 to $38 per operated capture hour; that premium is small against a wasted training run.

Why Single-View Data Caps Policy Performance

A single-view dataset gives the policy exactly one projection of the world per timestep, so any state the camera cannot resolve is unlearnable. Occlusion is the obvious case: the gripper approaches, the object disappears behind it, and the policy is now acting on memory and hope. Less obvious is spatial grounding. Language-conditioned instructions like “put the mug behind the plate” require scene geometry a close-cropped wrist view never encodes.

The RoboMimic study (arXiv:2108.03298) quantified the general point years ago: on identical demonstrations, changing the observation space, including camera views, materially changed imitation learning outcomes. Observation design is a first-order variable. Yet teams routinely treat it as fixed plumbing while sweeping learning rates for a month.

We see the ceiling directly in our ablations at DexSet. On tabletop pick-place and insertion tasks, moving from wrist-only to wrist plus one calibrated external view produced the largest single jump in success rate we have measured from any data intervention. Adding a second external view helped again on occlusion-heavy tasks. No optimizer change came close.

What the Big Datasets Already Decided

The major post-2023 collection efforts treat multi-view as a requirement, and that consensus is evidence in itself. DROID (arXiv:2403.12945) recorded roughly 76,000 Franka episodes across 564 scenes with two external ZED stereo cameras and a wrist ZED Mini on every single episode, extrinsics calibrated. The protocol did not permit single-view shortcuts, because the authors understood that scene diversity is worthless if the model cannot see the scene.

Ego-Exo4D (arXiv:2311.18259) went further for human demonstration data: over 1,200 hours of skilled activity captured simultaneously from Aria glasses and multiple stationary exocentric cameras, synchronized and calibrated, precisely so models can learn correspondences between first-person and third-person views. Anyone planning to pretrain robot policies on human video needs that pairing.

Open X-Embodiment (arXiv:2310.08864) is the counterexample that proves the rule. It aggregates over a million trajectories from 22 embodiments, and its camera configurations are heterogeneous: wrist-only here, one exo view there, different poses everywhere, extrinsics often missing. Teams pretraining VLAs on it spend real engineering time coping with viewpoint inconsistency. The lesson is not that aggregation is bad; it is that viewpoint standards are cheap at capture time and expensive to retrofit.

The Real Bottleneck Is Operational, Not Scientific

The reason most data is still single-view is not ignorance; it is that multi-view capture is an operations problem disguised as a shopping list. Buying three cameras takes an afternoon. Keeping their extrinsics valid, their clocks aligned, and their output QA’d across hundreds of sessions is the part that defeats teams.

Three costs dominate:

  • Calibration maintenance. Extrinsics drift when mounts get bumped, booms sag, or thermal cycles shift fixtures. Without per-session verification, drift silently corrupts weeks of data. Our gate is a 0.5 px reprojection error check on a ChArUco sweep at every session start.
  • Synchronization. Software timestamps drift across devices; PTP (IEEE 1588) or hardware trigger lines fix it, but only if someone engineers and monitors the sync path. We reject sessions with more than 10 ms cross-camera skew on manipulation work.
  • Storage and throughput. A 4-camera 1080p30 rig produces 0.8 to 1.5 TB per capture day in our pipelines. Multiply your single-view storage budget by three or four, then add QA review time.

None of this is research. All of it is why the bottleneck persists.

What Closing the Gap Costs

The honest comparison is single-view capture cost versus multi-view capture cost versus the cost of the training runs and engineering time the single-view ceiling wastes. Our benchmark numbers:

Item Single-View (Wrist or 1 Exo) Multi-View (2 Exo + Wrist) Delta
Rig build $1,500 to $2,500 $4,500 to $7,000 +$3,000 to $4,500 one-time
Operated capture $18 to $25 / hr $26 to $38 / hr +$8 to $13 / hr
Storage per capture day ~0.3 TB ~1.0 TB ~3x
Occlusion-heavy task ceiling Hard cap, architecture-independent Removed The point

For a 500-hour dataset, the multi-view premium lands around $4,000 to $6,500 in capture plus the one-time rig delta. One senior engineer spending six weeks fighting a data-imposed plateau costs more, and one full retraining run on data you have to recollect anyway costs far more. The full cost model, camera comparisons, and rig geometry options are in our pillar guide: The Complete Guide to Exocentric & Multi-View Data for Robot Learning.

How to Scale Multi-View Capture Without Drowning

Scaling multi-view data means industrializing the boring parts. The checklist we run internally:

  • Standardize one rig geometry (we default to DROID-style: two external stereo, one wrist) so calibration procedures and QA gates are identical across stations.
  • Gate every session on a two-minute calibration verification clip; reject on reprojection error > 0.5 px or sync skew > 10 ms.
  • Automate extrinsics logging into the episode metadata, so every frame carries its camera poses forever.
  • Budget storage at 3-4x single-view and decide codec and retention policy before capture starts, not after the first full disk.
  • Ablate camera count on your own tasks before scaling past three views; in our experience the fourth camera rarely earns its cost.

Run the Failure Analysis Before the Next Sweep

If your policy metrics have plateaued and your dataset is single-view, run the failure analysis before the next architecture sweep: tag failures by whether the target was visible at decision time. If occlusion dominates, the fix is capture. Book a demo and we will walk you through calibrated multi-view sample episodes from our production rigs, with the calibration and sync metadata included.

Request a Quote

Frequently Asked Questions

Why is multi-view data considered the bottleneck in physical AI?

Because model architectures and compute have outpaced data quality: policies trained on single-view episodes hit occlusion and spatial-grounding ceilings that no architecture change removes, and calibrated multi-view capture is operationally hard enough that most existing datasets never provided it.

In DexSet benchmarks, a DROID-style rig costs $4,500 to $7,000 versus $1,500 to $2,500 for single-view, and operated capture runs $26 to $38 per hour versus $18 to $25. Storage roughly triples.

Standardize one rig geometry across stations, gate every session on calibration and sync checks, embed extrinsics in episode metadata, budget storage at 3-4x single-view, and ablate camera count on your own tasks before adding a fourth view.

DROID enforced two external stereo views plus wrist across 76,000 episodes; Ego-Exo4D paired ego and exo video across 1,200+ hours for cross-view learning; RoboMimic showed observation space choices materially change imitation outcomes; Open X-Embodiment shows the integration cost when viewpoint standards are absent.

No. In our ablations the second view delivers the largest gain, a third helps on occlusion-heavy tasks, and a fourth is rarely distinguishable from noise while adding roughly 25 percent to storage and QA cost.

Case Study: How We Scaled Exocentric & Multi-View Data for a VLA Model

Posted on by Enoch Pakanati

We once shipped a pilot batch with an 18 millisecond wrist-camera clock offset, and this case study exists partly because of it. The mistake was ours: the wrist stream was timestamped off the arm controller instead of the rig’s PTP clock domain, invisible in playback, and it was the customer’s smoke-test training run at hour 20 that surfaced it. We fixed it, changed our rig standard, and kept the lesson. Vendor case studies that report zero mistakes are describing a project that never touched hardware.

This one covers eight weeks of work with a humanoid foundation model team (anonymized by agreement) whose manipulation policy had stalled on cluttered-scene tasks. The claim we will argue with the numbers below: their ceiling was informational, not architectural, and a disciplined multi-view capture operation, pilot batches, per-session calibration gates, one shared clock domain, is what removed it.

The team arrived with a specific complaint. Their policy trained fine, evaluated fine on open scenes, then dropped hard on cluttered tabletops. Their dataset: tens of thousands of teleoperated episodes, every one recorded from a single fixed camera. Failure review showed what you would expect. When clutter occluded the target from that one viewpoint during approach, the policy guessed.

They asked us for multi-view capture at a rate their internal rig could not hit: hundreds of hours within a quarter, calibrated, synchronized, and QA’d to a standard their ML leads could defend. What follows is how we built it, what it cost, what broke, and what the retrain showed. For the general framework behind these choices, the pillar guide is here: The Complete Guide to Exocentric & Multi-View Data for Robot Learning.

Key Takeaways – 600 hours of DROID-style multi-view episodes (two external stereo + wrist) delivered across 8 weeks on 4 parallel stations. – Per-session calibration gates rejected 4.1 percent of sessions; every rejection would have shipped corrupted extrinsics without the gate. – All-in capture cost landed at $31 per hour, inside our standard $26 to $38 benchmark range. – Retrained on matched episode counts, the customer’s occlusion-heavy split improved by double digits; open-scene performance held.

The Starting Point: Diagnosing a Data Ceiling

A data ceiling is a performance plateau caused by information missing from the training set rather than by model capacity. Confirming one is cheap and worth doing before any capture contract: tag evaluation failures by whether the target object was visible to the training viewpoint at decision time. On this team’s cluttered-scene split, occlusion-correlated failures dominated by a wide margin. No architecture sweep fixes pixels that were never captured.

The finding matches the public record. The RoboMimic study showed observation space design materially changes imitation outcomes on identical demonstrations (arXiv:2108.03298), and DROID’s authors considered multi-view important enough to mandate two calibrated external stereo views plus wrist across all 76,000 episodes (arXiv:2403.12945). We proposed the same geometry rather than inventing one.

The Rig Spec: Boring by Design

The capture spec is the contract between operations and the ML team, and ours fit on one page. Four identical stations, each with:

  • Two ZED 2i stereo cameras on rigid tripod mounts at roughly 45 degrees off the workspace centerline, 1 m from task center, opposite sides.
  • One wrist camera on the arm.
  • Extrinsics calibrated with a ChArUco target via OpenCV, cross-checked in Kalibr; camera-to-robot-base transform verified against known end-effector poses.
  • Sync via PTP-disciplined clocks, with a hard gate: cross-camera skew under 10 ms or the session does not ship.
  • A 20-second ChArUco verification sweep at every session start; reprojection error above 0.5 px blocks capture until recalibration.

Identical stations mattered more than any individual choice. One geometry means one calibration procedure, one QA script, one training data schema, and operators who can rotate between stations without retraining.

What Eight Weeks Actually Looked Like

Scaling capture is a throughput problem with a quality constraint, and the numbers tell the story better than prose:

Metric Value
Capture stations 4 (identical DROID-style geometry)
Calendar time 8 weeks
Delivered episodes ~58,000 across 41 task variants
Delivered hours (multi-view) 600
Sessions rejected at calibration/sync gate 4.1%
Episodes rejected at QA review 2.7%
Storage delivered ~38 TB (H.265, with per-frame extrinsics and sync metadata)
All-in operated cost $31 / hr

Weeks one and two ran at half throughput on purpose. We shipped a 20-hour pilot batch first so the customer’s ML team could confirm schema, load episodes into their LeRobot-based training stack, and run a smoke-test train before we committed the fleet. They caught the wrist-camera clock defect described at the top of this post: a consistent 18 ms offset from timestamping off the arm controller instead of the PTP domain. Catching that at hour 20 instead of hour 600 is the entire argument for pilot batches.

Throughput past the pilot came down to two decisions that had nothing to do with cameras. The first was task design: the customer’s 41 task variants were sequenced so that each station ran one object set per half-day block, which cut resets and scene changes to minutes instead of the constant churn you get when operators bounce between tasks. The second was operator rotation. Because all four stations shared one geometry and one procedure, any operator could run any station, and we scheduled captures to keep stations busy through breaks and calibration stops. Utilization across the fleet held near 85 percent of scheduled hours; on prior projects with heterogeneous rigs, we had struggled to hold 65.

The other failure worth naming: in week five, a boom mount on station three sagged after a fixture swap, and the morning verification sweep caught reprojection error at 1.3 px. The gate rejected the session, recalibration took 25 minutes, and no corrupted data shipped. Before we ran per-session gates, that class of drift used to surface weeks later as unexplainable training noise.

The Result: What the Retrain Showed

The customer retrained the same architecture on matched episode counts, single-view versus our multi-view data, which is the only comparison that isolates the data effect. On their occlusion-heavy cluttered split, success improved by double digits. On open scenes, performance held flat, confirming the ceiling had been informational, not architectural. Their engineers also reported a second-order win: with three calibrated views per episode, failure triage became visual inspection instead of guesswork, because someone could always see what happened.

We report ranges rather than their exact internal metrics by agreement, and we would flag any vendor who publishes a client’s precise evaluation numbers as a reason to negotiate confidentiality carefully. What we can say precisely is what the comparison controlled for: same architecture, same hyperparameters, same episode count, same evaluation protocol. The only variable was the data.

What We Would Repeat, and What We Changed

Three practices carried the project and are now standard on every DexSet engagement: pilot batches before fleet commitment, per-session calibration and sync gates with hard thresholds, and extrinsics embedded in every episode’s metadata rather than in a side document that drifts out of date.

One thing we changed afterward: we now put the wrist camera on the PTP domain from day zero, on every rig, because the 18 ms lesson generalizes. And we stopped quoting capture programs without a failure-tagging pass on the customer’s existing evaluation data first; twice since, that pass showed the bottleneck was not viewpoint at all, and we said so.

Test the Pattern on Your Own Failures

If your evaluation failures cluster around occlusion and your dataset is single-view, the pattern in this case study probably applies to you. Book a demo and we will walk through the pipeline with real sample episodes, calibration metadata included, and run the failure-tagging pass on your evaluation data before anyone talks about a contract.

Request a Quote

Frequently Asked Questions

How long does it take to scale a multi-view capture program?

In this engagement, 600 hours of calibrated multi-view data took 8 weeks on 4 parallel stations, including a deliberately slow 2-week pilot phase. Throughput scales roughly linearly with identical stations once the geometry and QA gates are standardized.

All-in operated capture landed at $31 per hour, inside DexSet’s standard $26 to $38 benchmark range for DROID-style rigs, plus one-time rig builds in the $4,500 to $7,000 range per station.

A 20-second ChArUco verification sweep at every session start, gated on reprojection error under 0.5 px, with extrinsics cross-checked in Kalibr and camera-to-base transforms verified against known end-effector poses. Sessions failing the gate are recalibrated before any capture ships.

Yes. Retrained on matched episode counts, the customer’s occlusion-heavy evaluation split improved by double digits while open-scene performance held, isolating viewpoint coverage as the binding constraint.

Because schema and sync defects are cheap at hour 20 and expensive at hour 600. The pilot here caught a wrist-camera clock offset of 18 ms that would otherwise have contaminated the full delivery.

The Complete Guide to Egocentric Data Collection for Robotics (2026)

Posted on by Sainath Gupta

3,670 hours. That is the complete Ego4D corpus, the largest first-person video dataset ever assembled (arXiv:2110.07058), and it amounts to roughly seven months of one person’s waking life. The robot foundation models expected to generalize across every kitchen, warehouse, and workbench on earth are drawing from a first-person data supply about that size, while their language-model cousins trained on trillions of tokens. The shelf is not thin. It is nearly bare, and teleoperation refills it at a few hundred action-labeled hours per rig per year.

The gap exists because robots perceive the world from their own body. A vision-language-action (VLA) model driving a humanoid needs to learn from footage that looks like what its head camera will actually see: hands entering the frame from below, objects at counter height, occlusions caused by the manipulator itself. That viewpoint is called egocentric, and until recently there was no scaled, systematic way to collect it. Our thesis, argued with numbers throughout this guide: egocentric capture is the only collection method that scales to foundation-model volumes, but it earns that scaling only when modality mix, viewpoint geometry, and annotation depth are derived from the training mechanism rather than from a hardware catalog.

This guide covers the full stack: what egocentric data collection for robotics actually is, the modalities that matter (mono, stereo, depth, IMU, gaze, hand pose), the hardware options from a $500 Quest 3 to Aria Gen 2 research glasses, how the data plugs into policy training, and what it costs per hour. We include the benchmark numbers we use internally at DexSet, because pricing opacity is the single biggest complaint we hear from buyers.

DexSet supplies egocentric, exocentric, teleoperation, mono, and stereo data to physical AI teams. We have built and rebuilt the capture rigs, the QA pipelines, and the annotation stacks described below, and most of the numbers in this guide come from our own production logs.

TL;DR: Key Takeaways – Egocentric data collection captures first-person visual and sensor streams from a camera mounted at the head or chest of a human (or robot), matching the viewpoint a robot policy will see at inference time. – It is the most scalable source of manipulation pretraining data: a human wearing glasses collects demonstrations 3 to 5 times faster than a teleoperator on a bimanual rig, at roughly one third to one half the cost per hour in our benchmarks. – Hardware ranges from ~$500 (Meta Quest 3, GoPro head mounts) to research-grade Aria Gen 2 glasses with calibrated multi-camera, IMU, eye tracking, and on-device machine perception. – Egocentric human video does not replace teleoperation; the strongest results (EgoMimic, co-training pipelines behind modern VLAs) combine both. The action gap between human hands and robot grippers is the core technical problem. – Our production benchmarks: raw egocentric capture runs $15 to $22 per hour; fully annotated (hand pose, object tracks, temporal segmentation) runs $30 to $40 per hour. Teleoperation runs $28 to $60 per hour depending on rig and task complexity.

What Is Egocentric Data Collection for Robotics?

Egocentric data collection for robotics is the practice of recording synchronized video and sensor streams from a first-person viewpoint, typically a head-mounted or chest-mounted camera worn by a human demonstrator, to train robot perception and control models. The defining property is viewpoint: the camera sees the scene the way an embodied agent sees it, with the demonstrator’s own hands and workspace in frame.

Three properties separate egocentric robotics data from ordinary first-person video:

  1. Sensor completeness. A YouTube cooking clip is RGB only. A robotics-grade egocentric recording carries calibrated camera intrinsics and extrinsics, IMU streams for ego-motion, and often stereo pairs or depth so that 3D structure can be recovered.
  2. Action recoverability. The footage must support extraction of what the hands did: 3D hand pose, object 6-DoF tracks, contact events. Without recoverable actions, egocentric video is only useful for representation pretraining, not policy learning.
  3. Task intent. Recordings are organized into episodes with defined start states, goals, and outcomes, mirroring how robot demonstration datasets like those in Open X-Embodiment are structured (arXiv:2310.08864).

The reference datasets here are Meta’s Ego4D, 3,670 hours of daily-life egocentric video across 74 locations (arXiv:2110.07058), and EgoExo4D, which pairs egocentric and exocentric views of skilled activities with dense annotations (arXiv:2311.18259). Both were built for video understanding research; robotics teams now treat them as the template for what scaled first-person capture looks like.

Egocentric vs. Exocentric: Why Viewpoint Determines Value

Exocentric data is footage captured from an external, third-person viewpoint, such as a tripod camera watching a workbench, while egocentric data is captured from the agent’s own point of view. The distinction matters because a policy trained purely on third-person views must solve an extra correspondence problem at deployment: mapping an external observation of a scene onto its own body frame.

The relationship chain that matters for buyers runs like this: egocentric human video teaches visuomotor priors, teleoperation (through systems like ALOHA, arXiv:2304.13705) provides robot-embodiment action labels, imitation learning consumes both, and VLA models such as OpenVLA (arXiv:2406.09246) and π0 (arXiv:2410.24164) sit at the top of the stack. EgoExo4D demonstrated why you often want both viewpoints of the same episode: the exocentric view disambiguates whole-body motion that the egocentric camera cannot see.

In our pipelines, paired ego-exo capture adds roughly 20 to 30 percent to per-hour cost (a second calibrated camera, cross-view sync, extra QA) and is worth it for whole-body humanoid work. For tabletop manipulation, egocentric plus a single fixed reference camera is usually sufficient.

Core Modalities in Egocentric Capture

A modality is one synchronized sensor stream within a recording, and the modality mix determines both the cost of capture and what training objectives the data can support. The five that come up in nearly every RFP we see:

Mono RGB. A single color stream is the cheapest to capture and the only modality most internet-scale pretraining uses. Sufficient for representation learning and video prediction, insufficient on its own for metric 3D.

Stereo RGB. Two horizontally offset cameras allow metric depth recovery through disparity. Stereo is the workhorse for manipulation because grasp points need metric accuracy. Devices like the Intel RealSense D435i and D455 provide hardware-synced stereo pairs plus an onboard IMU; the D455’s wider baseline (95 mm vs. 50 mm) improves depth accuracy at counter-to-room distances.

Depth. Active or computed depth gives per-pixel range directly. Active IR depth degrades in sunlight and on reflective surfaces, which is why most of our outdoor captures rely on passive stereo instead.

IMU. Accelerometer and gyroscope streams recover head motion, enabling ego-motion compensation and SLAM. Project Aria glasses carry two IMUs precisely because ego-motion is that important for downstream 3D reconstruction (projectaria.com).

Gaze and hand pose. Eye tracking (available on Aria) reveals attention targets before the hand moves, and 3D hand pose is the raw material for retargeting human demonstrations to robot grippers. These are the modalities that convert “video” into “demonstration.”

Hardware: The 2026 Egocentric Rig Landscape

An egocentric capture rig is the wearable hardware package (cameras, IMU, compute, mounting) used to record first-person data, and rig choice is the largest single driver of both data quality and program cost. The four setups we run or evaluate most often:

Rig Approx. Hardware Cost Sensors Video Spec (Typical Capture Config) Calibration Best For
Meta Quest 3 (Passthrough Capture) ~$500 Stereo RGB passthrough, IMU, hand tracking 1280×1280 per eye class, 30 fps effective capture Factory, limited access Budget hand-tracked demos, teleop UI doubling as capture
Aria Gen 2 Research Glasses Research program device (not retail) RGB + mono SLAM cameras, 2 IMUs, eye tracking, spatial mics, on-device hand tracking RGB up to 8 MP class, SLAM cams at high frame rate Full factory calibration + MPS services Research-grade egocentric corpora, gaze + hand pose at scale
GoPro Head/Chest Mount $350–$550 Mono RGB (wide FOV), IMU Up to 5.3K, we typically run 4K/60 or 2.7K/120 Self-calibrated (checkerboard) High-volume, low-cost mono capture; harsh environments
RealSense D435i/D455 Helmet Rig (Custom) $700–$1,200 built Stereo IR + RGB, active depth, IMU 848×480 depth at 90 fps or 1280×720 at 30 fps, RGB 1080p Manual, per-rig Metric depth for manipulation, sim-to-real alignment

Three field notes from running these at volume:

  • Quest 3 is underrated as a capture device because its hand tracking gives you approximate 3D hand pose for free, but passthrough capture access is constrained and image quality trails dedicated cameras.
  • Aria Gen 2 is the quality ceiling. Factory-calibrated multi-camera plus eye tracking plus machine perception services means far less post-processing on our side. Access runs through Meta’s research program rather than retail channels, which affects fleet scaling plans.
  • GoPro rigs win on ruggedness and unit economics. The cost is downstream: no depth, so you pay in annotation and 3D lifting compute instead of hardware.

How Egocentric Data Trains Robot Policies

Egocentric data enters robot learning through three mechanisms: representation pretraining, action retargeting, and co-training with robot demonstrations. Understanding which mechanism you are buying data for should drive every spec decision.

Representation pretraining. Visual encoders pretrained on large egocentric corpora like Ego4D transfer to manipulation tasks better than encoders trained on third-person or object-centric images, because the visual statistics (hands, near-field objects, ego-motion blur) match deployment. This is the lowest-risk use of egocentric data: mono RGB is enough, and annotation requirements are light.

Action retargeting. Human hand trajectories extracted from egocentric video are mapped onto robot end-effectors, turning passive video into pseudo-demonstrations. This requires recoverable 3D hand pose, which is why gaze-and-hand-instrumented devices matter. EgoMimic (arXiv:2410.24221) showed that egocentric human data captured on Aria glasses, combined with robot data, improves manipulation policies over robot data alone.

Co-training. Modern VLA training mixes robot episodes (teleop, in formats like the LeRobot dataset standard, github.com/huggingface/lerobot) with human egocentric episodes in one curriculum. The human data supplies breadth of scenes and objects; the robot data anchors the action distribution to the target embodiment. Cross-embodiment training in Open X-Embodiment established the pattern that heterogeneous data mixtures beat single-source datasets, and egocentric human video is the cheapest heterogeneity you can add.

The failure mode to respect: the embodiment gap. Human wrists have degrees of freedom robot grippers lack, human reach and eye height differ from most robot platforms, and human demonstrators exploit compliance no rigid arm has. Data collection protocols can shrink this gap (constrained grasps, robot-plausible motion instructions, matched camera height), and we bake those constraints into our capture scripts.

Collection Approaches Compared: In-House, Crowdsourced, Vendor

A collection approach is the operational model used to produce the data: who wears the rig, who designs the tasks, and who owns QA. Most teams land on one of three models, and the trade-offs are stable across every program we have run.

Approach Cost per Finished Hour (Our Benchmarks) Throughput Ramp Quality Control Where It Breaks
In-house Capture Team Typically well above vendor rates once salaries, rig fleet, and management overhead are loaded in; often roughly double Slow: 2–3 months to steady state Tight, iterative Scaling past ~10 collectors; hiring drag
Crowdsourced / Distributed Low headline rate, before rejection Fast but noisy Weak; rejection rates at or beyond the top of our 10 to 30 percent planning band are common in our audits Calibration, sync, task compliance
Specialist Vendor (DexSet Model) $15–$40 fully QA’d, annotation-dependent 2–4 weeks to first delivery Contractual, sampled + automated Task designs needing daily iteration with your researchers

The honest read: in-house wins when your task distribution changes weekly and researchers need to redesign protocols on the fly. A vendor wins when the task list is stable and the bottleneck is volume with consistent QA. Crowdsourcing looks cheap until you price the rejection rate and the engineering time spent triaging unsynced, uncalibrated footage.

Cost and Economics: What Egocentric Data Actually Costs

The cost of egocentric data is best expressed as dollars per finished, QA-passed hour at a defined annotation depth, because raw capture is a minority of total program cost. Competitors rarely publish numbers, so here are ours. These are current DexSet benchmark ranges, stated as typical figures we see across programs, not quotes:

Line Item Typical Range (Per Finished Hour) Notes
Raw Egocentric Capture (mono/stereo, IMU, episode structure) $15–$22 Collector time, rig amortization, upload, storage
+ Temporal Annotation (task/step segmentation, outcome labels) +$5–$8 Largely tooling-assisted
+ 3D Hand Pose + Object Tracks +$8–$12 The expensive layer; drives the $30–$40 fully-annotated figure
Paired Ego + Exo Capture +20–30% on capture line Second camera, cross-view sync, extra QA
Teleoperation (for comparison) $28–$60 Rig and task complexity dependent; bimanual fine manipulation sits at the top

Two planning rules of thumb from our production logs:

  • Budget 15 to 25 percent of hours for QA failure. Motion blur, dropped IMU packets, and off-task episodes are facts of life. Vendors should absorb this; if you collect in-house, plan for it.
  • Annotation depth should follow the training mechanism. If you are pretraining encoders, do not pay for hand pose. If you are retargeting actions, hand pose is the whole point. We regularly see RFPs over-specified by $10+ per hour because annotation depth was copied from a paper rather than derived from the training plan.

One more line item buyers forget: storage and delivery. Stereo capture at 30 fps with IMU sidecars generates roughly 50 to 120 GB per hour depending on resolution and compression, so a 5,000-hour corpus is a few hundred terabytes before derivatives. Cloud egress on a corpus that size is real money, which is why we quote delivery format and transfer method inside the per-hour price rather than as a surprise on the final invoice. Ask any vendor to do the same.

At these rates, a 5,000-hour egocentric corpus with full annotation lands between $150K and $200K. The equivalent volume via bimanual teleoperation would run $140K to $300K and take 3 to 5 times as long on a comparable rig fleet, which is the arithmetic behind the current industry shift toward human egocentric pretraining with a smaller teleop fine-tuning set.

Case Study Proof: 4,000 Hours for a Humanoid VLA Team

A humanoid foundation model team came to us with a stalled co-training experiment: their teleop corpus was high quality but topped out near 400 hours, and scaling it 10x on their own rigs would have taken most of a year. We scoped a 4,000-hour egocentric program across kitchen, warehouse shelving, and assembly-bench task families, captured on stereo rigs at camera heights matched to their robot’s head frame, with hand pose and object tracks on the 30 percent of hours their researchers flagged as retarget-critical.

Delivery ran 14 weeks. Their team reported that co-training on the mixed corpus improved task success on unseen-object manipulation evaluations relative to their robot-only baseline, consistent with the direction published in EgoMimic-style co-training work. The full breakdown of task families, QA gates, and the capture protocol is in the case study blog this pillar links to below.

Related reading from this series: – Why Egocentric Data Collection for Robotics Is the Biggest Bottleneck in Physical AI – Comparing Egocentric Data Collection Approaches: Pros, Cons and Costs – Case Study: Scaling Egocentric Data Collection for a VLA Model – 5 Hidden Challenges in Egocentric Data Collection

Download: The Egocentric Data RFP Template

An RFP template turns this guide into a procurement tool: it lists the 40 questions we believe every buyer should ask a data vendor, covering rig specs, calibration evidence, sync tolerances, QA sampling methodology, annotation rubrics, pricing structure, and data licensing. We built it from the RFPs we answer, including the questions we wish more buyers asked. Download it, delete our name from the header if you like, and send it to every vendor on your shortlist including us.

Put These Numbers to Work

If you are scoping an egocentric data program this quarter, two options. Download the RFP template and pressure-test every vendor with it, or book a 30-minute demo and we will walk you through sample episodes from our stereo and Aria-class rigs, including the QA reports we ship with every batch. Either way, you leave with real numbers instead of a sales deck.
Request a Quote

Frequently Asked Questions

What is egocentric data collection for robotics?

Egocentric data collection for robotics is the recording of synchronized first-person video and sensor streams (RGB, stereo, depth, IMU, gaze, hand pose) from head- or chest-mounted rigs, structured into task episodes, to train robot perception and manipulation models.

Egocentric data captures a human performing tasks with their own hands from a first-person camera, while teleoperation data captures a robot performing tasks under human control, with exact joint-space action labels. Egocentric data is cheaper and faster to scale; teleoperation data matches the robot embodiment exactly. Most modern VLA pipelines use both.

Common rigs include Meta Quest 3 (~$500, stereo passthrough and hand tracking), Aria Gen 2 research glasses (calibrated multi-camera, IMU, eye tracking), GoPro head or chest mounts ($350 to $550, mono RGB), and custom helmet rigs built around Intel RealSense D435i or D455 stereo depth cameras.

In DexSet’s benchmarks, raw QA-passed egocentric capture runs $15 to $22 per hour, and fully annotated data with hand pose and object tracks runs $30 to $40 per hour. Teleoperation data runs $28 to $60 per hour for comparison.

No. Egocentric human video scales pretraining and improves generalization, but the embodiment gap between human hands and robot grippers means policies still need robot-embodiment data (teleoperation or autonomous rollouts) for reliable control. Research such as EgoMimic supports combining both.

It depends on the mechanism: encoder pretraining benefits from thousands of hours of lightly annotated video, while retargeting pipelines often start showing gains with hundreds of hours of densely annotated, task-matched capture combined with a robot demonstration set.