Two teams approached us in the same quarter with nearly identical VLA data programs. The first sent a two-page acceptance spec and an RFP that demanded numbers, then ran a paid pilot before signing with anyone. The second compared three price sheets and took the lowest. Half a year later, the first team’s deliveries were entering training runs the day they arrived; the second team was still bisecting a training regression that traced back to a sync tolerance no contract had ever specified. Same budget class, same architecture, divergent quarters. The difference was not capture quality. It was procurement.
The uncomfortable part is that capture itself has scaled. ALOHA-class rigs are reproducible from public documentation (arxiv.org/abs/2304.13705). Open X-Embodiment pooled over a million episodes across 22 embodiments (arxiv.org/abs/2310.08864). DROID collected 76,000 teleop episodes across 13 institutions (arxiv.org/abs/2403.12945). The hardware and process knowledge exist. What has not scaled is the buying side: most teams still purchase demonstration data with less rigor than they apply to a laptop refresh.
This post makes the case that procurement is now the binding constraint, shows what the gap costs in numbers, and gives you the four artifacts that close it. It condenses the full Robotics Data Buyer’s Playbook, which includes the complete scorecard and RFP question bank.
We see this from the vendor side at DexSet. The buyers who send a real spec get better data at better prices than the buyers who send a budget and a hope, because a real spec lets us commit to numbers instead of hedging against unknowns.
Key Takeaways
- Capture capacity has commoditized; vendor selection has not. Weak procurement is now the most common cause of stalled VLA data programs.
- The cost of a bad pick is measured in usable hours: a cheap vendor at 70 percent yield can cost more per usable hour than a pricier one at 90 percent, once rework is priced in.
- Latent defects (sync error, calibration drift) are invisible in previews and only surface in training, which is why sample-based buying fails.
- The fix is procedural, not heroic: a spec-first RFP, a 10-criterion weighted scorecard, a red flags list, and a 50-hour paid pilot.
What Makes Procurement the Bottleneck
The procurement bottleneck is the delay and waste created when robot data purchasing decisions are made without a specification, a scoring method, or a pilot, forcing quality problems to surface downstream during training. It shows up as three concrete failure patterns.
Pattern one: the invisible defect. Robot data defects are latent. A 40 ms sync offset between camera frames and joint states will not appear in video playback, but it corrupts the state-action mapping your imitation learning policy depends on. Teams that buy on sample previews systematically miss this class of problem, then spend weeks bisecting training regressions that were purchased, not coded.
Pattern two: the spec vacuum. When the buyer has no written acceptance spec, every vendor quote describes a different product. One vendor’s “hour of manipulation data” is stereo egocentric plus wrist cameras with dense annotations; another’s is a single 720p exocentric mono stream. Comparing their prices is meaningless, and procurement stalls in clarification loops that a two-page spec would have prevented.
Pattern three: the format tax. Deliveries arrive in whatever the vendor uses internally: undocumented HDF5, half-converted rosbag2, a proprietary container. Your engineers pay the conversion cost into LeRobot or RLDS on every delivery. In our experience that tax runs 15 to 25 percent of program cost when format compatibility was never contracted.
The Cost of Buying Badly, in Numbers
The cost of weak procurement is best expressed as price per usable hour, which is the delivered price divided by the fraction of hours that pass your acceptance spec. The sticker price is the number vendors compete on; the usable-hour price is the number your training run experiences.
| Scenario | Sticker price | Usable-hour yield | True price per usable hour | 1,000-hour program cost (usable basis) |
|---|---|---|---|---|
| Cheapest bid, no pilot | $32/hr | 70% | $45.71 | $45,710 |
| Mid bid, sample-only check | $40/hr | 82% | $48.78 | $48,780 |
| Higher bid, passed 50-hr pilot | $46/hr | 90% | $51.11 | $51,110 |
| Cheapest bid after rework and triage time | $32/hr + engineering time | 70% | $55–$60 effective | $55,000–$60,000 |
Read the last row carefully. The cheapest vendor is the most expensive one once you price the engineering time spent triaging failures and patching training mixes, and that is before counting the schedule slip, which no spreadsheet captures but every roadmap feels. These are typical ranges from our benchmarks; your yields will vary, which is exactly why you measure them in a pilot instead of assuming them.
The Fix: Four Artifacts, Not More Meetings
The fix for the procurement bottleneck is a set of four reusable artifacts that convert vendor selection from judgment calls into measurements. Each one is small. Together they remove the guesswork that creates the bottleneck.
- A spec-first RFP. Write the acceptance spec (cameras, sync tolerance under 10 ms, task list, annotation rubric, delivery format) before contacting anyone. Then send 15 questions that force numeric answers: committed usable-hour yield, audit accuracy, rig counts, per-hour pricing by task complexity.
- A weighted scorecard. Ten criteria, weights summing to 100, with data quality SLAs, modality coverage, and calibration/sync spec carrying 39 points between them. Two independent scorers, reconciled. Automatic disqualification for weak answers on quality, calibration, or licensing.
- A red flags list. Some behaviors end the conversation regardless of score: pricing only after discovery calls, refusal of paid pilots, proprietary-only formats, no consent documentation, broad data reuse rights.
- A 50-hour paid pilot. Production rigs, production operators, and four predefined numbers: usable-hour yield (85 percent or higher), annotation audit accuracy (97 percent or higher on a 5 percent sample), policy success delta on a fixed eval set, and loader time into your LeRobot or RLDS pipeline (one engineer-day or less).
The full versions, including the complete scorecard weights and all 15 RFP questions, are in the Robotics Data Buyer’s Playbook.
How to Tell If Procurement Is Your Bottleneck
A procurement bottleneck diagnosis is a check of where data-program time actually goes, and it takes one honest hour with your delivery logs. Run through five questions:
- Can anyone state your acceptance spec from memory, or point to the document? If the spec lives in tribal knowledge, every vendor conversation is renegotiating it implicitly.
- What fraction of delivered hours reached a training run last quarter? If nobody tracks this number, assume it is worse than you think; teams that start measuring usually find 15 to 30 percent of purchased hours never trained anything.
- How long does a delivery take to enter the pipeline? More than a few hours of engineer time per batch means you are paying the format tax on every delivery.
- Could you defend your current vendor choice to your board with numbers? A scorecard produces that defense as a byproduct. A demo-based decision cannot.
- When did a data defect last cost you a training debugging cycle? If the answer is “this quarter,” the defect entered through procurement, not through capture.
Two or more uncomfortable answers means the bottleneck is upstream of your training code, and the fix below is cheaper than the symptom.
Scaling the Playbook Across Programs
Scaling this process means running the same artifacts on every purchase rather than reinventing evaluation per deal. The spec becomes a living document versioned alongside your model releases. The scorecard weights shift as your risks shift: teams early in data collection weight modality coverage higher; teams scaling a proven recipe weight throughput and SLAs higher. Pilot results accumulate into an internal vendor database, which is the closest thing this industry has to a track record. After three or four cycles, vendor evaluation drops from weeks of meetings to days of scoring, and that is the bottleneck removed.
Frequently Asked Questions
Why is procurement a bigger bottleneck than capture in physical AI?
Because capture methods are now well documented (ALOHA, DROID, Open X-Embodiment) while buying practices are not. Latent defects like sync error survive sample review and surface in training, so unstructured purchasing converts directly into stalled training runs.
What is price per usable hour?
Delivered price divided by the fraction of hours passing your acceptance spec. A $32/hr vendor at 70 percent yield costs $45.71 per usable hour, which is why sticker-price comparisons mislead.
How long should a robotics data pilot run?
Around 50 paid hours. That volume exposes process problems (calibration drift, operator variance, QA gaps) while keeping a failed pilot cheap: typically $1,500 to $4,500 at market rates.
What should an RFP require for delivery format?
Native delivery in LeRobot, RLDS/TFDS, or documented HDF5, with MCAP or rosbag2 as ROS 2 options. Proprietary-only delivery is a disqualifier because it adds conversion cost to every delivery and blocks independent audits.
