One house rule on our collection floor reliably surprises visitors: no episode ships the day it is collected, however clean it looks on the monitor, because every episode waits for the overnight QA gate first. The rule exists because the visible challenges in teleoperation, buying rigs, hiring operators, picking a data format, are the ones teams plan for. The hidden challenges surface at episode 5,000, when the dataset is half-collected, the deadline is fixed, and the training run just exposed a flaw that has been silently poisoning every episode since week one.
That is the thesis of this post: the failures that actually kill teleoperation datasets are invisible in any single episode and only show up in aggregate, so the fix is instrumentation at ingest, not heroics at training time. Each problem looks like a minor engineering detail until it compounds. A 40-millisecond timestamp offset is invisible in a demo video. Spread across 10,000 episodes, it is the difference between a 70 percent policy and a 45 percent one.
Below are the five failures we see most often across the programs we run and audit, each with the fix we actually use. If you are new to the space, start with the complete teleoperation data collection guide and come back.
Key Takeaways – Timestamp skew between 50 Hz actions and 30 fps cameras is the top QA rejection cause we see. Gate at 33 ms, at ingest. – Operator churn is a 3x throughput event, not an HR footnote: trained operators produce 25 to 40 usable episodes per hour versus 8 to 12 for novices. – Reset drift quietly narrows your state distribution; randomization zones fix it. – QA discovered at training time is loss; QA at ingest is feedback. Expect 10 to 30 percent rejection either way, so choose when you find out. – Format decisions are nearly irreversible after 10,000 episodes. Adopt LeRobot conventions on day one.
1. Timestamp Skew: The 50 Hz vs 30 fps Trap
Timestamp skew is the misalignment between a robot’s high-rate action stream and its lower-rate camera streams, and it is the single most common reason episodes fail our QA gates. Actions log at 50 Hz; cameras deliver 30 fps; USB buffering, clock drift between machines, and software timestamps assigned at receipt rather than exposure can push streams 30 to 100 ms apart. A policy trained on skewed data learns that consequences precede actions. It will move confidently and miss.
The fix. Hardware timestamps at the sensor wherever the camera supports it; a single shared monotonic clock otherwise; and an ingest gate that rejects any episode with estimated inter-stream skew above one camera frame (33 ms). We also log a clap-style sync event at the start of each shift as a cheap ground-truth check. Action chunking in ACT-style training tolerates per-step jitter, but no algorithm recovers from systematic offset (arxiv.org/abs/2304.13705).
2. Operator Churn: The Throughput Cliff
Operator churn is the loss of trained teleoperators before their throughput has paid back their ramp cost. The ramp curve is steep: 8 to 12 usable episodes per hour in the first two weeks, 25 to 40 after certification. Every departure resets a station to novice yield for two to three weeks, which means a floor with 20 percent monthly churn is quietly running at two-thirds of its rated capacity.
The fix. Treat operators as skilled technicians, because they are. Certification gates with pay progression, per-operator quality dashboards (people improve what they can see), 45-minute session blocks on VR rigs to manage fatigue, and rig ergonomics worth caring about. Leader-follower stations retain operators better than SpaceMouse benches for the simple reason that the work feels like skill rather than struggle.
3. Reset Drift: The Shrinking State Distribution
Reset drift is the gradual narrowing of initial-state diversity as operators unconsciously converge on the fastest reset, placing objects in the same pose, episode after episode. Nobody notices because every individual episode passes QA. The policy trains beautifully, evaluates beautifully on the collection table, then fails the moment an object appears five centimeters left of its usual spot. This is how teams end up with 10,000 episodes of one initial state.
The fix. Marked randomization zones on the workspace, task cards that specify pose ranges, and a distribution monitor in the QA pipeline that plots initial object positions per task per week. When the scatter tightens, the floor lead hears about it the next morning. Datasets like DROID earned their generalization value from scene diversity, 564 scenes across 76,000 episodes, not from episode count alone (arxiv.org/abs/2403.12945).
4. Silent QA Loss: Finding Out at Training Time
Silent QA loss is the gap between episodes collected and episodes actually usable, discovered only when someone trains on the data weeks later. Rejection is not the problem; 10 to 30 percent rejection is normal and we publish those numbers. The problem is timing. Rejection found at ingest becomes operator feedback and a corrected process by tomorrow. Rejection found at training time is a hole in the dataset, a slipped deadline, and no way to know which of the last six weeks went wrong.
The fix. Nightly automated gates on every episode: sync skew, dropped frames, joint-trajectory jerk thresholds, gripper event sanity, and success-label verification against a sampled human review. Route failures to the specific operator and rig. Our floors run this as a standing morning report, and it is the highest-ROI engineering month a data program can spend.
5. Format Lock-In: The Decision You Make Once
Format lock-in is the practical irreversibility of episode schema decisions once collection is underway, because re-processing tens of thousands of episodes costs more than most teams will ever approve. Missing per-episode metadata, camera intrinsics never recorded, action spaces defined ambiguously (commanded versus measured joint positions, unlabeled): each omission is trivial to fix on day one and a migration project at episode 20,000.
The fix. Adopt community conventions before the first production episode: LeRobot dataset format for training and sharing (github.com/huggingface/lerobot), HDF5 or ROS 2 bags at capture (docs.ros.org), both commanded and measured joint states recorded, camera intrinsics and extrinsics stored per rig per calibration, and schema versioning from v0. Fifteen minutes of schema review now, or fifteen days of migration later.
How to Audit Your Own Pipeline This Week
A pipeline audit is a half-day exercise that estimates your exposure to all five hidden challenges using data you already have. No new tooling required, in most cases.
Pull ten random episodes from last month and compute the maximum camera-to-control timestamp offset per episode; anything over 33 ms means challenge one is live in your dataset right now. Ask your floor lead for usable-episodes-per-hour by operator tenure; if you cannot get the number, that absence is itself the finding. Scatter-plot initial object positions for your highest-volume task across weeks one and four of collection; visibly tightening scatter is reset drift in progress. Check when rejection is discovered: if the answer involves a training run, you have silent QA loss by definition. And finally, hand an episode file to an engineer who has never seen your schema and time how long it takes them to load actions, states, and calibrated camera streams; over thirty minutes predicts format pain at scale.
Every one of these checks has caught a live problem on programs we have audited, including our own. The five challenges are not exotic failure modes; they are the default state of a pipeline that nobody has audited yet.
The Checklist
| Hidden challenge | Detection signal | Fix | Cost of ignoring |
|---|---|---|---|
| Timestamp skew | Skew estimate > 33 ms at ingest | Hardware timestamps + sync gate | Policies that act early/late; top rejection cause |
| Operator churn | Throughput below 20 eps/hr floor-wide | Certification, pay progression, ergonomics | Floor runs at ~2/3 capacity |
| Reset drift | Initial-state scatter tightening week over week | Randomization zones + distribution monitor | Policy memorizes one layout |
| Silent QA loss | Rejection discovered at training time | Nightly ingest gates + operator feedback | 10 to 30% of budget wasted invisibly |
| Format lock-in | Schema questions at training time | LeRobot conventions + schema v0 | Migration project at episode 20k |
Frequently Asked Questions
What is the most common QA failure in teleoperation data?
Inter-stream timestamp skew, typically between 50 Hz action logs and 30 fps camera streams. We gate at 33 ms of estimated skew and it remains the top rejection reason on new rigs.
How much teleoperation data fails quality checks?
Expect 20 to 30 percent rejection during the first two weeks on a new task or rig, settling to 10 to 15 percent in a mature pipeline. Budget the buffer into episode targets.
Why do teleoperation-trained policies fail on new object positions?
Usually reset drift: operators converge on repeated initial states during collection, so the dataset covers a narrow state distribution. Randomization zones and initial-state monitoring prevent it.
When should QA run on teleoperation episodes?
At ingest, nightly, never later. Rejection found at ingest becomes next-day operator feedback; rejection found at training time is unrecoverable loss weeks after the fact.
What data format should a new teleoperation program use?
LeRobot dataset conventions for training and sharing, with HDF5 or ROS 2 bags at capture. Record both commanded and measured joint states, camera calibration per rig, and version the schema from the start.