Last April, a batch that had passed all six automated checks in our pipeline dragged a customer’s held-out task success below baseline, and it took a probe-policy run to show us why: every episode was individually clean and collectively near-identical. That incident restated a lesson we keep relearning, and it is the thesis of this post. The dangerous failures in robot training data are not the obvious ones. Nobody ships a dataset full of black video; the black frames get caught. What ships is the episode where the wrist camera runs 40 milliseconds behind the control loop, the “successful” grasp that slipped one second after the label cutoff, the thousand episodes that are individually flawless and collectively identical. Your training curve inherits all of it.
These failures stay hidden because each one lives in a blind spot of a standard QA method. Human reviewers cannot see sub-frame timing. Scripts cannot judge whether a success label matches reality. Per-episode checks of any kind cannot see distribution. Teams run one method, feel covered, and get burned by the failure class that method structurally cannot detect.
Below are the five hidden challenges we hit most often running teleoperation and egocentric collection at DexSet, why each evades standard checks, and the specific fix and threshold we use in production. Between rig operations and QA rotation, I have personally traced each of these from a bad training curve back to root cause at least once.
Key Takeaways – The costliest data failures pass at least one standard QA method; coverage comes from layering methods with disjoint blind spots. – Sub-frame desync, label-cutoff errors, and clean-but-narrow batches are the three we see damage training most often. – Each challenge below has a concrete production fix: a check, a threshold, or a process change. – Full pipeline context, costs, and the complete check catalogue live in our pillar guide.
Challenge 1: Action-Observation Desync That Survives Human Review
Action-observation desync is a timing offset between what the robot saw and what it did, and offsets of one to three frames are invisible to a human watching playback. At 30 fps, a 2-frame lag is 66 milliseconds. Reviewers pass it all day. A policy trained on it learns to act on stale observations, then acts early or late at deployment, and the error compounds along the trajectory in exactly the way the imitation learning literature predicts (Ross et al., arXiv:1011.0686).
The fix: cross-correlate motion energy between camera streams and joint velocities per episode, per stream pair, and reject anything beyond 1 frame at 30 fps. Do not check streams only against a master clock; we have caught wrist cameras drifting against the control loop while both stayed individually monotonic. In one vendor audit, this single check rejected an entire 350-hour delivery that had passed the vendor’s own review.
Challenge 2: Success Labels That Lie at the Margins
Success-label error is a mismatch between the annotated outcome and the real outcome, and it concentrates on near-miss episodes: grasps that slip after the labeling cutoff, tasks completed while toppling adjacent objects, pours that land 90 percent in the target. Annotators under throughput pressure round these up. Since most pipelines filter training data on the success flag, every rounded-up failure goes straight into the imitation set with a “copy this” sign on it.
The fix: audit label accuracy, do not assume it. Re-judge a random 5 to 10 percent of episodes against ground-truth video with a second annotator, require agreement at or above 0.90, and hold audited success-label accuracy at 95 percent or better before a batch ships. Define success operationally in the rubric (end-state held for N seconds, no non-target disturbance) rather than leaving “success” to intuition.
Challenge 3: Clean but Narrow Batches
Distribution narrowness is a batch of episodes that all pass per-episode QA while collectively covering a sliver of the state space: one base pose, one approach direction, one lighting condition, one operator’s habits. Every episode is fine. The batch teaches a policy that generalizes to almost nothing, and no per-episode check of any kind can flag it. DROID went to 564 scenes precisely because scene narrowness had limited earlier corpora (arXiv:2403.12945).
The fix: two layers. Cheap layer: monitor per-batch dispersion statistics on approach poses, object positions, and episode durations, and alert when variance collapses. Expensive layer: policy-in-the-loop validation, training a small probe policy per batch and comparing held-out success against baseline. The probe run is the only test that measures generalization directly. We budget $1 to $4 per hour for it and it has caught every narrow batch that dispersion stats missed.
Challenge 4: Operator Recovery Segments Left in the Data
Recovery contamination is the flailing between a mistake and its correction: approach, miss, retreat, reposition, retry. Operators recover constantly, and that is fine; unsegmented recovery in the training set is not. The policy learns oscillation as a valid strategy. The ACT authors’ handling of noisy human demonstrations exists partly because this class of noise is endemic to teleoperation (arXiv:2304.13705), but architecture tolerance has limits that curation should not lean on.
The fix: track intervention rate (interventions per episode) as a first-class metric during collection, keep it under 0.3, and give operators a one-button “mark and re-run” instead of mid-episode recovery for short tasks. For long-horizon tasks where retakes are expensive, have reviewers segment recovery spans so training can mask or downweight them. Rising intervention rate is also your earliest rig-degradation alarm; it moves days before yield drops.
Challenge 5: Cross-Source Convention Mismatches
Convention mismatch is a silent disagreement between data sources on units, frames, or schemas: radians versus degrees, camera-frame versus base-frame actions, differing gripper conventions, success defined three different ways. Each source passes its own QA. Pooled, they hand the model a contradiction to memorize. This is the problem Open X-Embodiment spent enormous effort normalizing across 22 embodiments (arXiv:2310.08864), and it recurs in miniature inside any two-vendor buy.
The fix: a cross-source audit before pooling. Verify units by checking value ranges against physical limits, replay actions in a kinematic viewer to confirm frame conventions, validate camera intrinsics against delivered calibration, and run one probe-policy batch per source before ever training on the pool. Standardized formats such as LeRobot’s remove schema-level disagreement (github.com/huggingface/lerobot), which helps, but format agreement does not imply semantic agreement.
Sequencing the Fixes
Fixing all five challenges at once is neither necessary nor realistic, so sequence by damage per dollar. Start with the desync check: it is a day of scripting, it runs on everything you have already collected, and in our experience it produces the single largest one-time recovery of trust in a corpus. Second, the label audit, because it needs only a second annotator and a written rubric, and its findings usually justify the rest of the program to whoever owns the budget.
Recovery segmentation and intervention tracking come third; they require operator workflow changes, which take weeks to land. Distribution monitoring and probe policies come fourth, once you have batches worth testing. Cross-source audits slot in whenever a second vendor appears, and not a delivery later. Teams that run this sequence typically go from no formal QA to a working three-tier pipeline in one quarter, without pausing collection.
The Checklist
| Hidden challenge | Why standard QA misses it | Production fix | Threshold |
|---|---|---|---|
| Action-observation desync | Invisible to human review | Per-pair cross-correlation check | Reject >1 frame @30 fps |
| Lying success labels | Scripts cannot judge outcomes | Double-annotator audit sample | ≥95% audited accuracy, ≥0.90 agreement |
| Clean-but-narrow batches | Per-episode checks cannot see distribution | Dispersion stats + probe policy per batch | Probe success ≥ running baseline |
| Recovery contamination | Looks like normal demonstration | Intervention tracking, segmentation | <0.3 interventions/episode |
| Convention mismatches | Each source passes its own QA | Cross-source audit before pooling | Per-source probe run required |
Next Step
These five fixes slot into the three-tier pipeline documented in our complete guide: Data Quality & QA for Robotics Datasets, which includes the full automated check catalogue, cost benchmarks, and the downloadable QA Scorecard. If one of these failure modes sounds like your training curve, book a pipeline review.
Frequently Asked Questions
Why does my robot policy underperform even though the data passed QA?
Because standard QA methods have disjoint blind spots. The usual suspects are sub-frame action-observation desync, mislabeled near-miss successes, and distributionally narrow batches, none of which a single-method QA process catches.
What desync threshold should robot datasets enforce?
Reject episodes with cross-stream offset beyond 1 frame at 30 fps (33 ms), checked per stream pair rather than only against a master clock.
How accurate do success labels need to be?
We hold batches to 95 percent audited accuracy with inter-annotator agreement at or above 0.90, because success flags gate what enters imitation training.
How do you detect a distributionally narrow dataset?
Monitor dispersion statistics on approach poses and object positions per batch, and train a small probe policy per batch against a held-out baseline. Only the probe run measures generalization directly.
What is a healthy intervention rate during teleoperation collection?
Under 0.3 interventions per episode in our production experience. A rising rate is an early warning of rig degradation or operator fatigue, appearing days before yield drops.
