Semiconductor fabs settled the argument between volume and quality decades ago: no fab reports wafer starts to its board, it reports yield, the fraction of known-good die that survive inspection. Robot learning is having the same argument about training data right now, and most teams are still on the wrong side of it. They plan, buy, and report raw hours, while in our production experience 10 to 30 percent of those hours should never reach a training set. That is the thesis of this guide: the unit of progress in robot learning is the usable hour, and QA is the yield discipline that produces it.
The gap between raw and usable exists because robotics data fails in ways that image or text data does not. A demonstration can look fine to a human scrolling thumbnails while hiding a 40 millisecond action-observation desync that teaches the policy to act on stale states. Timestamps drift. Cameras drop frames. Operators recover from mistakes in ways that a naive success label hides. None of this shows up in a dataset card that reports only hours and episode counts.
This guide gives you a complete, operational QA system for robot learning data: the three-tier pipeline we run at DexSet, the exact automated checks worth implementing first, the quality metrics that actually predict downstream policy success, and honest numbers on what QA costs and what it returns.
DexSet collects egocentric, exocentric, and teleoperation data for VLA and humanoid foundation model teams. Every claim about pipelines, rejection rates, and costs below comes from running that operation, not from theory.
TL;DR: Key Takeaways – Data quality and QA for robotics datasets is the systematic verification that demonstrations are temporally consistent, physically plausible, correctly labeled, and useful for policy learning. – Bad demonstrations do not average out. Imitation learning suffers compounding errors under distribution shift, so a small fraction of corrupted episodes can drag policy success down disproportionately. – The workable structure is a three-tier pipeline: automated checks on every episode, human spot review on a sample, and policy-in-the-loop validation on candidate batches. – Expect to reject 10 to 30 percent of raw collected hours. Budget an extra $5 to $12 per hour for QA on top of collection cost. – Track usable-hour yield, not raw hours. It is the single most predictive operational metric we know of.
What Is Data Quality & QA for Robotics Datasets?
Data quality and QA for robotics datasets is the set of processes that verify robot demonstration data is temporally synchronized, physically plausible, correctly annotated, and distributionally suited to the policy that will train on it. In practice it spans three layers: signal integrity (are the sensor streams intact and aligned), semantic correctness (do labels like task success or object annotations match reality), and learning utility (will this episode help or hurt the policy).
The distinction from generic data QA matters. A duplicated image in a classification set wastes a training step. A demonstration with a mislabeled success flag actively teaches a manipulation policy that a failed grasp is the goal state. Robotics data QA is closer to flight data validation than to image dataset cleaning: the data encodes closed-loop behavior, and errors in it become errors in behavior.
The entity chain runs like this: teleoperation rigs (ALOHA-style leader-follower arms, VR rigs, exoskeletons) produce demonstrations; demonstrations feed imitation learning methods such as ACT and diffusion policies; those methods underpin VLA models like OpenVLA and π0. Quality problems injected at the rig propagate down the entire chain.
Why Garbage Demos Poison Policies
A small fraction of bad demonstrations degrades imitation learning far more than the same fraction of bad samples degrades supervised learning, because policies act on their own predictions. Ross, Gordon, and Bagnell formalized this in the DAgger paper: behavior cloning errors compound over a trajectory, and the learner drifts into states the expert never visited, where its errors grow further (arXiv:1011.0686). A corrupted demo does not just add noise to one gradient step. It plants behavior that the policy then executes, generating out-of-distribution states at inference time.
The ALOHA/ACT work makes the same point from the practitioner side. Zhao et al. built ACT specifically to cope with the compounding-error problem in fine manipulation and note that human demonstration data is noisy and stochastic, which is exactly why they model action chunks and train on carefully collected demonstrations (arXiv:2304.13705). Our first-hand version of this lesson: on internal ablations across bimanual manipulation tasks, training runs that included our rejected-episode pool consistently underperformed runs trained on the cleaned subset, even when the cleaned subset had fewer total hours. Less data, higher quality, better policy. We have seen that pattern often enough that we now treat it as an operating assumption rather than a hypothesis.
Foundation-model teams learned the analogous lesson in language and vision: curation beats raw scale once you pass a minimum data volume. Open X-Embodiment showed that pooling heterogeneous robot datasets helps, but the authors had to normalize action spaces, camera conventions, and annotation schemes across 22 embodiments to make the pooled data usable at all (arXiv:2310.08864). Heterogeneity without QA is not diversity. It is noise with extra steps.
The DexSet 3-Tier QA Pipeline
A three-tier QA pipeline layers cheap automated checks over every episode, human judgment over a sample, and actual policy training over candidate batches, so that each tier catches what the previous one cannot. This is the structure we run in production at DexSet, and each tier exists because we found failure modes the tier above it missed.
| Tier | What it is | Coverage | Catches | Typical cost share |
|---|---|---|---|---|
| Tier 1: Automated checks | Scripted validation on every stream of every episode | 100% of episodes | Signal integrity failures: desync, dropouts, limit violations, outliers | ~20% of QA budget |
| Tier 2: Human spot review | Trained reviewers scoring sampled episodes against a rubric | 10-20% sample, 100% of flagged episodes | Semantic failures: wrong success labels, sloppy strategy, occluded views | ~50% of QA budget |
| Tier 3: Policy-in-the-loop validation | Train a small policy on the candidate batch, evaluate on held-out tasks | Per batch, not per episode | Learning-utility failures: distribution gaps, systematic operator habits | ~30% of QA budget |
Tier 1 runs at ingest, before an episode is ever eligible for delivery. It is deterministic, fast, and boring, which is the point. Roughly 60 to 70 percent of all rejections in our pipeline happen here.
Tier 2 exists because automation cannot judge intent. A reviewer watching a pick-and-place episode can see that the operator knocked over an adjacent object and the success label ignored it. We sample 10 to 20 percent of passing episodes plus everything Tier 1 flagged as marginal, and we track inter-reviewer agreement to keep the rubric honest.
Tier 3 is the tier most vendors skip because it costs GPU time. We train a lightweight ACT-style policy on each candidate batch and compare held-out task success against the running baseline. When a batch that passed Tiers 1 and 2 still drags evaluation success down, the usual culprit is distributional: an operator who always approaches objects from the same side, or a scene setup that quietly narrowed object diversity. Policy-in-the-loop validation is the only tier that measures what buyers actually care about, which is whether the data trains better robots.
The Automated Check Catalogue
An automated check catalogue is the concrete list of scripted validations run on every episode at ingest, each with a hard threshold that triggers rejection or human review. These six checks catch the majority of signal-integrity failures we see in teleoperation and egocentric data:
| Check | What it detects | Our threshold | Typical share of Tier 1 rejections |
|---|---|---|---|
| Timestamp monotonicity | Clock resets, message reordering, NTP jumps | Any non-monotonic timestamp in any stream | 10-15% |
| Camera dropout / black frames | USB bandwidth saturation, cable faults, exposure failures | >0.5% dropped frames or any black-frame run >3 frames | 20-30% |
| Joint limit violations | Bad kinematic calibration, encoder faults, retargeting bugs | Any commanded or measured position outside URDF limits | 5-10% |
| Action-observation desync | Buffering drift between control and camera streams | Cross-stream offset >1 frame at 30 fps (>33 ms) | 15-25% |
| Gripper state mismatch | Commanded gripper state disagrees with measured aperture | Disagreement >200 ms outside contact events | 10-15% |
| Episode length outliers | Stalled operators, forgotten recordings, premature stops | Outside 3 MAD of the per-task episode length distribution | 10-15% |
Egocentric and stereo collections need three additions to this catalogue. IMU dropout detection matters because a head-mounted stream with gaps in its inertial track cannot be gravity-aligned or stabilized downstream; we reject on any IMU gap longer than 100 ms. Stereo calibration drift matters because rigs get bumped: we verify epipolar error on sampled frame pairs and flag episodes above 1 pixel. Exposure flicker matters for outdoor and mixed-lighting egocentric capture, where auto-exposure hunting produces frame sequences that destabilize vision encoders; a rolling luminance variance check catches it cheaply.
Two implementation notes from production. First, run desync checks per stream pair, not just against a master clock; we have seen wrist cameras drift against the control loop while both stayed monotonic. Second, joint limit checks should validate against the same URDF the training stack uses. If you consume data in the LeRobot format, its dataset tooling already enforces schema and per-episode metadata consistency (github.com/huggingface/lerobot), which is a useful floor, but format validity is not signal validity. A LeRobotDataset can be perfectly well-formed and still contain a desynced wrist camera.
Quality Metrics That Actually Predict Policy Success
Quality metrics for robotics data are the small set of numbers that correlate with downstream policy performance, as opposed to vanity numbers like raw hours collected. Four metrics have earned a permanent place on our dashboards:
Usable-hour yield. The fraction of collected hours that survive all three QA tiers. Our fleet-wide range is 70 to 90 percent depending on task difficulty and rig maturity, meaning rejection rates of 10 to 30 percent. When a buyer compares two data vendors on price per hour, the real comparison is price per usable hour, and a 20-point yield gap swamps a 15 percent price difference.
Intervention rate. Interventions per episode during collection, where a supervisor or the operator aborts and recovers. Rising intervention rates are a leading indicator: they show up days before yield drops, usually signaling rig degradation or operator fatigue.
Annotation agreement. For labeled data, inter-annotator agreement on categorical labels and IoU on spatial ones. We require IoU at or above 0.85 on object bounding boxes and at or above 0.90 agreement on success labels before a batch ships. Below those thresholds, label noise starts to dominate the signal you are paying annotators to create.
Demo success labeling accuracy. Audited accuracy of the success flag itself, measured by re-judging a random sample against ground-truth video. No single label in the dataset carries more weight, because most imitation pipelines filter on it. A 5 percent error rate in success labels means 5 percent of your “successes” are failures that the policy will imitate faithfully.
| Metric | Healthy range (our benchmarks) | Red flag |
|---|---|---|
| Usable-hour yield | 70-90% | <70% sustained over a week |
| Intervention rate | <0.3 per episode | >0.5 and rising |
| Bounding box IoU | ≥0.85 | <0.80 |
| Success label accuracy | ≥95% on audit | <90% |
What QA Costs, and What It Returns
Robotics data QA typically adds $5 to $12 per collected hour on top of collection cost, and in our experience it is the highest-ROI line item in a data budget. The range depends on annotation density and how much Tier 3 compute you run. Against teleoperation collection costs that commonly run $28 to $60 per hour depending on rig and task complexity, QA is a 10 to 30 percent overhead.
| Line item | Typical cost (our benchmarks) | Notes |
|---|---|---|
| Teleop collection | $28-$60 / hr | Rig type and task complexity are the main drivers |
| Tier 1 automated QA | $1-$2 / hr | Mostly amortized engineering plus compute |
| Tier 2 human review | $3-$6 / hr | Scales with sampling rate and rubric depth |
| Tier 3 policy validation | $1-$4 / hr | GPU time, amortized per batch |
| Total QA overhead | $5-$12 / hr | 10-30% on top of collection |
The return side is harder to state as a universal number, so we will state it as first-hand experience: across our internal comparisons, policies trained on QA-filtered batches beat policies trained on equivalent raw batches on held-out task success, and the gap is largest on long-horizon and fine manipulation tasks where compounding errors bite hardest. The cheap intuition: if 20 percent of your hours are poisoning training, every dollar of the collection budget is doing 80 cents of work at best. Paying 15 percent overhead to recover that is not a cost. It is arbitrage.
There is also a second-order saving. Rejected episodes analyzed at ingest become rig fixes. Half of our automated-check categories map directly to a physical remediation (cable, sync board, calibration routine), so QA feedback shortens the loop between “data is bad” and “rig is fixed” from weeks to days.
Case Study: QA at Scale for a Humanoid Foundation Model Team
A humanoid foundation model team came to us after a 1,000-hour collection effort from mixed vendors produced training runs that were unstable batch to batch. Some deliveries helped. Some measurably hurt. They had no way to tell which was which before spending GPU budget.
We re-ingested the full corpus through the three-tier pipeline. Tier 1 rejected 14 percent of hours outright, dominated by action-observation desync (one vendor’s rig had a consistent 2-frame camera lag) and black-frame runs. Tier 2 review flagged another 9 percent, mostly success-label errors on episodes where the humanoid completed the task but dropped or displaced non-target objects. Tier 3 flagged one vendor’s entire delivery as distributionally narrow: nearly every episode approached the workspace from an identical base pose.
Net usable-hour yield across the corpus: 74 percent. The team retrained on the cleaned 740 hours and reported more stable evaluation curves and higher held-out success than the original 1,000-hour runs, with the added benefit that their per-vendor yield numbers became the basis for renegotiating two contracts. The full breakdown is in the companion post: Case Study: How We Scaled Data Quality & QA for a VLA Model.
QA Across Heterogeneous Data Sources
Cross-source QA is the extra validation layer required when a training corpus mixes embodiments, rigs, or vendors, because consistency failures between sources are invisible to per-episode checks. Open X-Embodiment is the canonical illustration: 22 embodiments, dozens of institutions, and a paper’s worth of normalization work on action spaces, camera frames, and success conventions before the pooled data trained anything (arXiv:2310.08864). DROID hit related issues at the scene level, collecting across 564 scenes precisely because narrow scene distributions had limited earlier corpora (arXiv:2403.12945).
If you are buying from multiple vendors, add three cross-source checks to the catalogue: unit and frame-convention audits (we have caught radians-versus-degrees mismatches that per-episode checks pass), camera intrinsics verification against delivered calibration files, and per-source Tier 3 runs so a weak source cannot hide inside a pooled batch. For more on where teams get burned, see 5 Hidden Challenges in Robotics Data QA and the approach comparison in Comparing Robotics Data QA Approaches.
Build or Buy: Who Should Run Your QA
The build-or-buy decision for robotics data QA turns on volume and on whether QA feedback needs to reach your own rigs. Teams collecting on internal hardware should build at least Tier 1 themselves, because the automated checks double as rig telemetry: a spike in gripper mismatches is a maintenance ticket, and outsourcing that signal means losing it. The scripts are a few engineer-weeks; the thresholds in this guide are a working starting set.
Teams buying data face a different calculus. You cannot fix a vendor’s rig, so the value of in-house QA is verification, not remediation. The minimum viable setup is Tier 1 checks at ingest plus a per-delivery probe-policy run, which together cost far less than one disputed delivery. Whatever you build, put the same thresholds in the vendor contract, so a failed check is a contractual event rather than an argument. Our RFP scorecard below exists because buyers kept asking us for exactly that language.
Download: The Robotics Data QA Scorecard
We condensed this guide into a one-page scorecard: the six automated checks with thresholds, the four dashboard metrics with healthy ranges, and a vendor-evaluation rubric you can paste into an RFP. Teams use it two ways: to audit an internal collection pipeline, or to force per-usable-hour pricing transparency from data vendors. It pairs well with the bottleneck analysis in Why Data QA Is the Biggest Bottleneck in Physical AI.
Next Step
If your policy metrics have plateaued and you suspect the data, the fastest diagnostic is an audit, not another collection order. Download the Robotics Data QA Scorecard, or book a pipeline review with our data operations team and we will run a sample of your corpus through the three-tier pipeline and show you your real usable-hour yield.
Frequently Asked Questions
What is data quality and QA for robotics datasets?
It is the systematic verification that robot demonstration data is temporally synchronized, physically plausible, correctly labeled, and useful for policy training. It spans automated signal checks, human review of semantic labels, and validation by training policies on candidate data.
How much of a robotics dataset typically gets rejected in QA?
In our production pipelines, 10 to 30 percent of raw collected hours fail QA, depending on task difficulty and rig maturity. Mature rigs on simple tasks sit near 10 percent; new rigs on long-horizon bimanual tasks can exceed 25 percent.
How much does robotics data QA cost?
Our benchmarks put full three-tier QA at $5 to $12 per collected hour, or roughly 10 to 30 percent overhead on typical teleoperation collection costs of $28 to $60 per hour.
Why do bad demonstrations hurt imitation learning so much?
Because policies act on their own predictions, errors compound along a trajectory and push the robot into states absent from training data, where performance degrades further. This distribution-shift dynamic is formalized in the DAgger paper (Ross et al., arXiv:1011.0686). A mislabeled or desynced demo teaches behavior, not just noise.
What automated checks should run on every episode?
At minimum: timestamp monotonicity, camera dropout and black-frame detection, joint limit validation against the training URDF, action-observation desync (reject beyond 1 frame at 30 fps), gripper command-versus-state mismatch, and episode length outlier detection.
Is the LeRobot format enough to guarantee data quality?
No. LeRobot tooling enforces schema and metadata consistency, which is a useful floor, but a well-formed dataset can still contain desynced streams, wrong success labels, or narrow distributions. Format validity is necessary, not sufficient.
