Key takeaways
- Teleoperation turns operator skill into training data — but only if the pipeline is synchronized, protocol-driven, and QA’d.
- Operator interventions and failures are a free, high-value source of edge-case data. Log them on purpose.
- Bad teleoperation data gets imitated. Filtering low-quality episodes matters as much as collecting them.
Teleoperation is one of the most direct ways to generate robot demonstration data — a person drives the robot, and the robot learns from what they did. But a sloppy setup produces data a policy will faithfully imitate, mistakes and all. Here’s how to build a teleoperation data pipeline that produces training-ready data. See the teleoperation playbook for the deeper reference and the pillar for context.
Step 1 — Choose the teleop interface and robot
Match the interface to the task. VR controllers and leader-follower arms give the dexterity that fine manipulation needs; a gamepad may suffice for coarser tasks. Choose the interface for the embodiment you’ll deploy, and remember that cross-embodiment data widens reuse.
Step 2 — Instrument multi-sensor synchronization
Log RGB, depth, proprioception (joint and end-effector state), and the operator’s control commands — all on one clock. Timestamp accuracy is not optional: a few milliseconds of desync between camera and control turns a good demonstration into label noise.
Step 3 — Define the task protocol and operator briefing
Write a repeatable protocol: start state, task definition, success criteria, reset procedure. Brief operators so episodes are consistent, and capture consent — teleoperation with people or in real facilities carries the same responsible-data obligations as any capture.
Step 4 — Log interventions and failures
When an operator takes over, corrects, or fails, that moment is exactly the edge case your autonomous policy will struggle with. Tag interventions and failures explicitly — they feed the data flywheel and are far cheaper than staging edge cases from scratch.
Step 5 — QA and filter episodes
Review for desync, jerky or low-quality control, and annotation noise. Filter episodes that would teach bad behavior. Imitation learning is only as good as its worst-frequent demonstration; a quality gate here is worth more than raw volume.
Step 6 — Package, version, and deliver
Export to your training formats, version the dataset like code, and retain provenance so results are reproducible and eval sets don’t leak. Now the pipeline is repeatable — run it on a cadence, not once.
Next Step
dexset stands up synchronized, QA’d teleoperation pipelines with intervention logging, task-level annotation, and versioned delivery.
Frequently Asked Questions
How many teleop episodes per task?
Enough to cover the task’s variation and failure modes; coverage, not a fixed count.
Is teleoperation better than human video?
They’re complementary — human video is strong for pretraining, teleoperation gives you robot-grounded action data for fine-tuning.
