You are three weeks from a milestone demo and the policy still drops the mug the moment the gripper crosses in front of it. The wrist camera looks perfect in the replay viewer, the demonstrations are clean, and none of it helps. The model never saw the scene from anywhere else, so the instant its one viewpoint goes blind, so does the policy.
The problem exists because most teams start with the camera that is easiest to mount, not the camera set that answers the questions their model will ask at inference time. A single egocentric or wrist view gives you fine-grained contact detail and nothing else: no scene context, no occlusion recovery, no spatial grounding for language-conditioned tasks. Fixing that after you have collected 400 hours of single-view teleoperation is expensive. Fixing it before you start costs a tripod and a calibration session.
The thesis of this guide is simple: camera geometry is a first-order training decision, not plumbing, and calibrated multi-view capture is the cheapest performance intervention most manipulation teams have not yet made. We will argue it three ways: with the published record (DROID, Ego-Exo4D, RoboMimic), with our own capture benchmarks and ablations, and with a worked customer example. This guide covers what exocentric and multi-view data actually is, which camera and rig configurations the major research datasets use, how to calibrate and synchronize multiple cameras without corrupting your dataset, and what all of it costs. The numbers come from our own capture operations at DexSet, where we run multi-camera teleoperation and human demonstration rigs daily for VLA and humanoid foundation model customers, plus the public dataset cards and papers we cite throughout.
TL;DR – Exocentric data is third-person footage of a robot or human performing a task; multi-view data captures the same episode from two or more calibrated, time-synchronized viewpoints at once. – The strongest public evidence for multi-view capture comes from DROID (two external stereo cameras plus a wrist camera on a Franka arm) and Ego-Exo4D (paired egocentric and exocentric video of skilled human activity). – Our benchmarks: a production-grade multi-view rig costs $2,500 to $12,000 to build, and calibrated multi-view capture runs $20 to $50 per hour depending on camera count, sync requirements, and QA depth. – The second camera delivers most of the benefit. In our ablations, going from one external view to two produced the largest jump in policy success; the fourth camera added little beyond storage cost. – Calibration drift and time sync are where multi-view datasets quietly die. Budget for recalibration checks every capture session, not every capture month.
What Is Exocentric Data for Robot Learning?
Exocentric data is visual data recorded from a third-person viewpoint that observes the robot or human demonstrator and the workspace from outside the body performing the task. A camera on a tripod behind the workbench, an overhead camera above a bin-picking cell, and a shoulder-height camera watching a humanoid fold towels are all exocentric views. The defining property is that the camera pose is independent of the actor’s motion.
Egocentric data is the complement: footage from the actor’s own perspective, such as smart glasses on a human demonstrator or a head-mounted camera on a humanoid. Wrist cameras sit in between; they move with the arm but do not share the actor’s gaze. Most serious manipulation stacks end up wanting both. The exocentric view supplies global scene context and object relationships, and the egocentric or wrist view supplies contact-level detail during grasps.
For a model, the practical difference shows up in failure modes. Policies trained only on exocentric views struggle with precise insertion because the interesting pixels are small and far away. Policies trained only on wrist views fail whenever the gripper blocks the object or the task requires reasoning about anything outside a 30 cm bubble. This is why datasets built for general-purpose policies pair the two rather than choosing.
What Is Multi-View Data?
Multi-view data captures a single episode from two or more cameras with known relative poses and aligned timestamps. Two properties separate a true multi-view dataset from a pile of videos that happen to point at the same table: calibrated extrinsics (each camera’s position and orientation relative to a shared frame, typically the robot base) and synchronization (frames across cameras correspond to the same instant, ideally within a few milliseconds).
Both properties are load-bearing. Without extrinsics, you cannot fuse views geometrically, project actions between frames, or train models that reason about 3D structure. Without sync, a policy learns from image pairs that show slightly different world states, which injects label noise you can never remove afterward. We reject capture sessions at DexSet when cross-camera timestamp skew exceeds 10 ms on manipulation tasks, because we have watched that skew turn into unexplainable policy failures downstream.
Multi-view is not the same as stereo. A stereo camera such as the ZED 2i is one viewpoint with two lenses on a fixed 12 cm baseline, built to estimate depth. A multi-view rig places whole cameras (mono, stereo, or RGB-D) at meaningfully different poses around the workspace. You can and often should combine them: two stereo cameras at different poses give you multi-view and per-view depth at once, which is exactly the configuration DROID chose.
What the Research Record Shows
The strongest public evidence for multi-view capture comes from three sources, and they agree with each other more than most robotics literature does.
DROID (arXiv:2403.12945, dataset) collected roughly 76,000 teleoperated episodes, about 350 hours, on Franka arms across 13 institutions and 564 distinct scenes. Every episode records two external ZED stereo cameras plus a wrist-mounted ZED Mini, with calibrated extrinsics. The authors made multi-view a hard requirement of the collection protocol, not an optional extra, because scene diversity only pays off if the model can see the scene.
Ego-Exo4D (arXiv:2311.18259, project page) is the largest paired ego-exo resource: over 1,200 hours of video of skilled human activity, captured simultaneously from Aria glasses on the participant and four or more stationary exocentric cameras, with camera calibration and time sync across the full rig. It exists because ego-only and exo-only datasets kept failing at cross-view tasks like translating “what I see” into “what the coach sees.” Robot learning inherits the same problem when transferring human video priors onto robot viewpoints.
Open X-Embodiment (arXiv:2310.08864, Hugging Face) aggregates over one million trajectories from 22 robot embodiments, and its per-dataset cards read like a natural experiment in camera configuration: some sources are wrist-only, some single-exo, some multi-view. Teams consuming it for VLA pretraining consistently report the friction of heterogeneous viewpoints, which is itself an argument for capturing calibrated multi-view from day one rather than harmonizing after the fact.
On the modeling side, the RoboMimic study (arXiv:2108.03298) found that observation space design, including which camera views feed the policy, materially changes imitation learning outcomes on the same demonstrations. Camera choice is a first-order hyperparameter, not plumbing.
Camera Hardware: What to Put on the Tripod
The camera decision is a trade between depth quality, shutter behavior, sync options, and price. These are the three units we deploy most, with the specs that actually matter for robot data capture.
| Spec | Intel RealSense D455 | Stereolabs ZED 2i | Luxonis OAK-D |
|---|---|---|---|
| Type | Active IR stereo RGB-D | Passive stereo + neural depth | Stereo + RGB, on-device compute |
| Depth baseline | 95 mm | 120 mm | 75 mm |
| Max RGB mode | 1280x800 @ 30 fps | 2208x1242 @ 15 fps (1080p @ 30) | 4K RGB @ 30 fps (12 MP sensor) |
| Depth range (practical) | 0.6 to 6 m | 0.5 to 20 m (neural) | 0.7 to 8 m |
| Shutter | Global (depth), rolling RGB on some SKUs | Rolling | Global (OV9282 stereo pair) |
| IMU | Yes | Yes (9-DoF, barometer) | Yes (on most variants) |
| Sync options | External sync pin, multi-cam sync | Timestamp-based, no genlock | Hardware trigger via GPIO |
| Street price | ~$420 | ~$500 | ~$250 |
| Where it wins | Tabletop manipulation, tight spaces | Longer range, outdoor, mobile robots | Budget rigs, embedded preprocessing |
Two practical notes from our rigs. First, rolling shutter plus fast arm motion produces skewed geometry that quietly degrades any 3D supervision; if your task involves dynamic motion, weight global shutter heavily. Second, the D455 sync pin and the OAK-D hardware trigger let you drive frame capture from a shared signal, while the ZED relies on timestamp alignment, which is fine at 30 fps tabletop speeds and marginal for high-speed tasks.
Rig Geometry: Where the Cameras Go
Rig geometry is the arrangement of camera poses around the workspace, and it matters as much as the cameras themselves. The configurations below cover nearly everything we build.
Two external + wrist (the DROID pattern). One camera at roughly 45 degrees over each shoulder of the workspace, 0.8 to 1.2 m from the task center, plus a wrist camera. This is our default for tabletop manipulation. It gives occlusion recovery (when one external view is blocked, the other usually is not), stereo-of-stereos geometry for 3D checks, and contact detail from the wrist.
Overhead + front. An overhead camera looking straight down disambiguates object layout for pick-and-place and bin tasks; a front camera at torso height captures approach trajectories. Overhead mounts need rigid fixturing. A camera on a boom arm that sags 2 mm over a week of sessions will silently invalidate your extrinsics, which is one of the failure modes we cover in 5 Hidden Challenges in Exocentric & Multi-View Data.
Ego + exo paired (the Ego-Exo4D pattern). Glasses or head-mounted camera on the demonstrator plus stationary exocentric cameras. This is the configuration to choose when your training strategy includes human video pretraining, because it gives you the cross-view correspondence needed to transfer human priors to robot viewpoints.
Ring or arc arrays (4 to 8 cameras). Necessary for full-scene reconstruction, humanoid whole-body capture, or world-model training data. Expensive in sync engineering and storage, and in our experience unnecessary for most single-arm manipulation targets.
Calibration and Synchronization: The Unglamorous Core
Calibration is the process of estimating each camera’s intrinsics (lens and sensor parameters) and the rig’s extrinsics (relative camera poses, plus camera-to-robot-base transforms). Synchronization is making frame timestamps agree across cameras. Neither is hard the first time. Keeping both true across hundreds of capture hours is the actual job.
The standard toolchain is mature and free. OpenCV handles single and stereo calibration with checkerboard or ChArUco targets. Kalibr handles multi-camera and camera-IMU calibration, which you want the moment your rig has more than two cameras or a moving ego camera. ROS 2 ships camera_calibration in image_pipeline for in-place intrinsic calibration on live topics.
For sync, there are three tiers. Software timestamping (each camera stamps frames against system clocks) is free and drifts. PTP, the IEEE 1588 Precision Time Protocol, disciplines device clocks over Ethernet to sub-millisecond agreement and is the right answer for GigE machine vision cameras. Hardware genlock or trigger lines drive every sensor’s shutter from one signal and are the only way to get true same-instant exposure, which matters at high frame rates or with fast motion. Consumer RGB-D cameras mostly give you the first tier plus, on some units, a trigger pin; plan your rig around that constraint rather than discovering it later.
One more transform matters for robot data specifically: camera-to-robot-base, often called hand-eye calibration. Extrinsics between cameras tell you how views relate to each other; the camera-to-base transform tells you how all of them relate to the robot’s action space, which is what lets you project end-effector trajectories into any view or supervise 3D policies in the base frame. We estimate it by touching the arm’s end effector to known target points visible to the cameras, then verify against commanded poses. Skip this and your multi-view dataset is geometrically consistent with itself and disconnected from the robot.
Our operational rule: capture a 20-second calibration verification clip (a ChArUco board swept through the shared view volume) at the start of every session, and gate ingestion on reprojection error staying under 0.5 px and cross-camera skew under 10 ms. It costs two minutes per session and has saved entire capture weeks.
Cost and Economics: What Multi-View Actually Costs
Multi-view capture costs break into rig build (one-time) and operated capture (per hour). The figures below are our own benchmarks from rigs we run at DexSet; treat them as typical ranges, not quotes.
| Configuration | Rig Build Cost | Operated Capture Cost | Typical Use |
|---|---|---|---|
| 1 external RGB-D + wrist (baseline) | $2,500 to $4,000 | $20 to $28 / hr | Prototyping, single-task policies |
| 2 external stereo + wrist (DROID-style) | $4,500 to $7,000 | $26 to $38 / hr | VLA training data, tabletop manipulation |
| Ego + 3 exo paired (Ego-Exo4D-style) | $6,000 to $9,500 | $32 to $44 / hr | Human demo capture, cross-view learning |
| 6-camera arc, hardware-triggered | $9,000 to $12,000 | $40 to $50 / hr | Humanoid whole-body, reconstruction, world models |
Rig build includes cameras, mounts and fixturing, sync hardware, a capture workstation, and the first calibration. Operated capture includes the operator, session calibration checks, QA review, and annotation-ready packaging; it excludes task design and annotation itself. Storage is the line item teams forget: a 4-camera rig at 1080p30 generates roughly 0.8 to 1.5 TB per capture day depending on codec, and multi-view triples or quadruples whatever single-view budget you had.
The marginal-value curve is the key economic fact. In our ablations on tabletop pick-place and insertion tasks, adding the second external view to a wrist-only setup produced the largest single improvement in policy success. The third camera helped mainly on occlusion-heavy tasks. The fourth was rarely distinguishable from noise while adding about 25 percent to storage and QA cost. Buy the second camera before you buy anything else; justify the fourth with an experiment, not a hunch. We break the per-configuration trade-offs down further in Comparing Exocentric & Multi-View Approaches: Pros, Cons & Costs.
Case Study: Multi-View at Scale for a VLA Team
A humanoid foundation model team came to us with a policy that plateaued at roughly 60 percent success on cluttered tabletop tasks, trained on single-view teleoperation data. Their failure analysis pointed at occlusion: success dropped sharply whenever the target object was blocked from the lone camera during the approach.
We rebuilt their capture around a DROID-style rig: two external stereo cameras plus wrist, hardware-checked calibration each session, sub-10 ms sync, and a QA gate on reprojection error. Over eight weeks we delivered several hundred hours of calibrated multi-view episodes across their task list. Retrained on the multi-view data with the same architecture and episode count, the policy’s success on the occlusion-heavy split improved by double digits, and their engineers stopped hand-labeling “camera blocked” failure cases entirely. The full pipeline detail is in How We Scaled Multi-View Data for a VLA Model, and the strategic version of the argument is in Why Multi-View Data Is the Biggest Bottleneck in Physical AI.
Downloadable: The Multi-View Rig RFP Scorecard
If you are evaluating data vendors or scoping an internal rig, the questions that separate real multi-view capability from marketing are specific: How are extrinsics verified per session? What is your cross-camera timestamp skew tolerance and how is it measured? What reprojection error gates ingestion? What is the storage format and per-hour deliverable size? We have packaged these into a one-page RFP scorecard with scoring weights you can hand to procurement. Download it below; it is the same rubric we hold our own rigs to.
Put This Guide to Work
If your current dataset is single-view and your policy failures cluster around occlusion or spatial grounding, the fix is a capture decision, not an architecture search. Download the Multi-View Rig RFP Scorecard to evaluate vendors or your own rig plan, or book a demo to see calibrated multi-view episodes from our production rigs, including sample data you can load the same day.
Frequently Asked Questions
What is exocentric data in robot learning?
Exocentric data is third-person visual data captured by cameras positioned outside the robot or demonstrator, observing the actor and workspace from fixed external viewpoints. It provides global scene context that egocentric and wrist cameras cannot, and it pairs with those views in most modern manipulation datasets.
How many cameras do I need for multi-view robot data?
Two calibrated external views plus a wrist camera cover most manipulation use cases; this is the configuration DROID used across 76,000 episodes. In our ablations the second external camera delivers the largest gain, while a fourth camera rarely improves policy success enough to justify its storage and QA cost.
What is the difference between stereo and multi-view data?
Stereo is one viewpoint with two lenses on a fixed short baseline, designed to estimate depth. Multi-view places separate cameras at meaningfully different poses around the workspace with calibrated extrinsics. A rig can be both, for example two ZED 2i stereo cameras at different positions.
How much does multi-view robot data collection cost?
Based on our benchmarks at DexSet, multi-view rig builds run $2,500 to $12,000 depending on camera count and sync hardware, and operated calibrated capture runs $20 to $50 per hour including session calibration checks and QA. Storage adds roughly 0.8 to 1.5 TB per capture day for a 4-camera 1080p30 rig.
How do you synchronize multiple cameras for robot datasets?
Three tiers: software timestamping (free, drifts), PTP / IEEE 1588 clock discipline over Ethernet (sub-millisecond, right for GigE machine vision cameras), and hardware genlock or trigger lines (true same-instant exposure). For tabletop manipulation at 30 fps, keep cross-camera skew under 10 ms and verify it every session.
Which datasets pair egocentric and exocentric views?
Ego-Exo4D is the largest, with over 1,200 hours of simultaneously captured ego (Aria glasses) and exo (stationary camera) video of skilled human activity, with calibration and sync across the rig. In robot-collected data, DROID pairs a wrist (near-ego) view with two calibrated external stereo views.
Do VLA models need multi-view training data?
VLA models benefit from multi-view data because language-conditioned tasks require scene-level grounding and manipulation requires contact-level detail, which no single viewpoint provides. The RoboMimic study showed observation space choices materially change imitation learning outcomes, and Open X-Embodiment’s heterogeneous camera setups are a recurring friction point for teams pretraining VLAs.
