Printing your timestamps is the checklist item almost every capture team skips, and skipping it is how a ten-thousand-episode dataset dies without a symptom. We watched it happen. The dataset looked perfect: clean video, complete joint logs, delivered on time. Then the client’s policy trained to a success rate barely above random, and three weeks of forensics traced it to a 60 ms clock offset between the wrist camera and the robot controller. Every action label in the dataset was stale by two frames, and a five-line timestamp check would have caught it before the first shift ended.
Failures like this stay hidden because capture hardware fails silently, and that is the argument this post makes: silent failure is the defining property of this hardware category, so the five failure modes below must be designed out before capture, not caught after it. A camera that stops recording is obvious. A camera whose clock drifts, whose calibration decays, or whose operator quietly degrades over an eight-hour shift produces data that looks fine and trains badly. Spec sheets describe none of this, because spec sheets describe devices, not fleets running sixty-hour weeks.
This post catalogs the five failure modes my QA team catches most often, what each one costs, and the specific fix for each. All five are hardware-level problems you can design out before your first episode is recorded; the device-by-device background lives in our full guide to data capture hardware for embodied AI.
I lead data quality at DexSet, where these checks run against every egocentric, teleoperation, and multi-view batch we deliver. The thresholds below are the ones on our own QA gates.
Key Takeaways – The five hidden challenges are clock drift, calibration decay, operator fatigue, offload bandwidth, and data format lock-in. None appear on a spec sheet. – Clock drift is the most expensive: tens of milliseconds of offset silently corrupts every action label in an episode. Our QA gate rejects episodes beyond 25 ms of cross-stream misalignment. – Calibration decays with heat and handling; we re-verify extrinsics on a fixed cadence (per shift for moved rigs) rather than trusting installation-day values. – Each challenge has a cheap hardware-stage fix and an expensive post-hoc one. The table below prices both.
Challenge 1: Clock Drift and Cross-Sensor Sync
Clock drift is the gradual divergence of independent device clocks, and it corrupts the observation-action alignment that imitation learning depends on. Consumer devices drift by seconds per day; even good oscillators disagree by milliseconds per hour. A dataset can pass every visual inspection while every frame’s action label describes what the robot did 60 ms later.
The fix is architectural. Prefer hardware with a native sync story: RealSense sync pins for camera arrays, Project Aria’s shared-clock VRS recordings for wearables, robot-side timestamping for teleop streams. Where hardware sync is impossible, inject a common observable event (a strobe, an audible clap, a scripted gripper snap) at episode boundaries and align in post. Our gate rejects any episode whose residual cross-stream misalignment exceeds 25 ms.
Challenge 2: Calibration Decay
Calibration decay is the slow invalidation of a rig’s intrinsic and extrinsic parameters through heat cycles, vibration, and handling. The extrinsics you measured at installation describe a rig that no longer exists three weeks later, once a camera mount has been bumped twice and a headset has been dropped once. Depth reprojection then smears by centimeters, which is fatal for grasp-point supervision.
The fix is cadence, not heroics. We re-verify extrinsics per shift on any rig that moves (headsets, handheld UMI-style grippers) and weekly on fixed studios, using a 60-second checkerboard or AprilTag routine baked into shift startup. Rigs that fail re-verification capture nothing until they pass. The routine costs about one percent of shift time and removes our second-largest historical rejection cause.
Challenge 3: Operator Fatigue as a Data Quality Variable
Operator fatigue is the degradation of demonstration quality over a shift, and it is a hardware property as much as a human one. A 600-plus-gram headset with an external battery pack produces different data at hour six than at hour one: slower corrections, shakier trajectories, more aborted grasps. The episodes still pass automated checks. The policy trained on them inherits the sloppiness.
The fix starts at procurement. Weigh the device; anything worn for full shifts should sit near action-camera weight (~150 g) rather than headset weight (500-650 g). For teleop, seated leader-arm designs in the ALOHA style (arxiv.org/abs/2304.13705) outlast free-space VR control on repetitive tasks. Then instrument for it: we track per-operator trajectory smoothness by shift-hour, and rotate stations when the curve bends.
Challenge 4: Offload Bandwidth and Storage Starvation
Offload bandwidth is the rate at which captured data moves off devices into storage, and when it is undersized the capture floor throttles itself. A six-camera RGB-D studio generates on the order of a terabyte per shift. Teams that budget for cameras but not for the offload path end a shift early to copy files, every day, forever, and consumer devices with slow card readers make it worse.
The fix is to size the pipe with the fleet: NVMe-based ingest per capture cell, hot-swappable media so devices never wait for uploads, and compression decisions made deliberately (lossless for depth and proprioception, visually lossless for RGB) rather than by device default. As a rule we budget storage-and-offload spend at 30 to 50 percent of camera spend; audits of stalled floors usually find it under 10 percent.
Challenge 5: Data Format Lock-In
Data format lock-in is dependence on a device’s proprietary recording format, and it converts every future pipeline change into a migration project. GoPro telemetry lives in GPMF tracks, ZED records SVO files, Aria writes VRS, every teleop stack invents an HDF5 layout, and none of them agree. The cost appears later, as engineer-weeks of converters and silent unit mismatches.
The fix is to normalize at ingest, on the same day the data is captured. Pick one canonical episode schema, the LeRobot dataset format (github.com/huggingface/lerobot) is the emerging community default, and treat device-native files as transient. Keep raw originals archived, but never let a training pipeline read a vendor format directly.
How to Audit Your Own Floor This Week
A capture audit is a one-week measurement pass that scores an existing pipeline against these five failure modes, and it needs no new hardware to run. The version we use internally:
- Day 1: Print timestamps. Sample 50 random episodes and compute the median camera-to-proprioception offset per episode. Anything past 25 ms goes on the findings list.
- Day 2: Re-run calibration on every rig and record the delta from stored values. Deltas above your reprojection tolerance mean decay is already in your data.
- Day 3: Plot per-operator trajectory smoothness by shift-hour. A visible bend after hour four is a hardware ergonomics finding, not a staffing one.
- Day 4: Time one full offload cycle per cell. Any cell where offload competes with capture time loses roughly a data hour per day.
- Day 5: Count the distinct file formats between device and training job. More than one canonical format plus archived originals means converter debt is accruing.
Most teams find at least three of the five on the first pass. That is not an indictment; it is the baseline this hardware category imposes on everyone.
The Five Challenges at a Glance
| Challenge | Silent Symptom | Cost If Ignored | Fix at Hardware Stage | Fix After the Fact |
|---|---|---|---|---|
| Clock drift | Policies underperform despite clean-looking data | Entire batches rejected or retrained | Sync pins, shared clocks, boundary strobes (~$0-200/rig) | Manual event alignment; often unrecoverable |
| Calibration decay | Depth smears, grasp labels off by centimeters | 10-20% rejection creep | Per-shift AprilTag re-verification (1% of shift time) | Re-calibration + partial batch salvage |
| Operator fatigue | Quality slides after hour 4 | Sloppy trajectories baked into the policy | Buy light (~150 g worn), seated leader arms, rotation | Filter late-shift episodes, lose volume |
| Offload bandwidth | Shifts end early to copy files | ~1 lost data hour/day per cell | NVMe ingest, hot-swap media (30-50% of camera spend) | Retrofit storage, disrupt operations |
| Format lock-in | Converter scripts multiply | Engineer-weeks per pipeline change | Normalize to one schema at ingest | Migration project across archived data |
Frequently Asked Questions
What is the most common hidden failure in robot data capture hardware?
Cross-sensor clock drift. Independent device clocks diverge by milliseconds per hour, silently misaligning observations and action labels. It is the failure mode least visible in review and most damaging in training.
How much clock misalignment is acceptable in a training dataset?
Our QA gates reject episodes with more than 25 ms of residual cross-stream misalignment for manipulation data. At 50-60 ms, action labels lag observations by one to two frames at 30 fps, which measurably degrades imitation learning.
How often should capture rig calibration be checked?
Per shift for any rig that moves or is worn (headsets, handheld grippers), and weekly for fixed camera studios. A 60-second AprilTag verification at shift start catches decay before it contaminates a batch.
Does operator fatigue really affect dataset quality?
Yes, measurably. On our floors, trajectory smoothness and correction latency degrade in the back half of shifts on heavy worn devices, which is why device weight and seated leader-arm ergonomics are QA concerns, not comfort perks.
What data format should robot capture pipelines standardize on?
Normalize everything to one canonical episode schema at ingest; the LeRobot dataset format is the strongest community default in 2026. Archive device-native files (GPMF, SVO, VRS) but never train from them directly.
Want these checks run for you?
Every DexSet delivery passes the five gates above before it ships. [Download Sample Data] with the QA report attached, or start with the full guide to data capture hardware for embodied AI.
