Grasp success on the team’s fixed eval suite dropped six points in a single week, and for two days nobody could explain it. The model had not changed. The training code had not changed. What had changed, it eventually emerged, was a data batch: one of their three vendors had silently rescaled gripper actions between deliveries. The team had money, talent, and a working humanoid platform. What stalled their VLA program was procurement: three data vendors, three delivery formats, no shared acceptance spec, and an ML team that had quietly stopped trusting incoming data. By their own estimate, roughly a quarter of purchased hours never reached a training run.
Nothing about that situation was unusual. It is what happens when data purchasing grows by accretion: a vendor added for a demo push here, another for a task category there, each with its own definitions of “hour,” “annotated,” and “calibrated.” The result is not one data pipeline but several, with the training team as the manual integration layer.
This post walks through how the team rebuilt procurement in one quarter using the four artifacts from the Robotics Data Buyer’s Playbook: a spec-first RFP, a weighted scorecard, a red flags screen, and a 50-hour paid pilot. The client is anonymized (a Series B humanoid startup; we do not name customers), but the process and the numbers are the real pattern we see when buyers switch from improvisation to protocol. The thesis this case argues: pilots run under production conditions predict production performance, and paper evaluation alone does not.
DexSet was one of the vendors evaluated in this process, so we saw the RFP, the pilot, and the contract from the receiving side. That is exactly the vantage point a buyer should want documented.
Key Takeaways
- Starting point: three vendors, three formats, ~74 percent average usable-hour yield, about one engineer-day of triage per delivery.
- The rebuild: one written acceptance spec, an RFP to six vendors, a 10-criterion weighted scorecard, paid 50-hour pilots for the top two.
- Pilot results diverged sharply: 90 percent yield with a positive policy delta versus 78 percent yield and a failed loader test.
- Twelve weeks later: yield held within two points of the pilot across the first 1,000 contracted hours, and per-delivery engineering time dropped from a day to under an hour.
The Starting Point: Accretion, Not Architecture
The team’s data supply had accreted rather than been designed: each vendor relationship was created for a short-term need, and none was ever re-evaluated. Vendor A delivered undocumented HDF5 with a schema that shifted between batches. Vendor B delivered rosbag2 that had to be converted and re-timestamped. Vendor C delivered mp4s plus CSV joint logs with no sync guarantees at all.
The measurable symptoms, from their own tracking:
- Average usable-hour yield around 74 percent across vendors, measured against the acceptance checks they eventually wrote down.
- Roughly one engineer-day per delivery spent on conversion and triage.
- Two training regressions in one quarter traced back to data batches, one to a sync drift issue, one to inconsistent gripper action scaling between vendors.
The trigger for change was mundane: a planning meeting where nobody could answer what an incremental 1,000 hours of manipulation data would actually cost, because nobody could say what fraction would be usable.
Step 1: Write the Spec Before Talking to Anyone
The acceptance spec is a short document that defines what a usable hour means for a specific program, and writing it was the most valuable week of the whole rebuild. Theirs fit on three pages: stereo egocentric plus wrist cameras at defined resolutions and frame rates, proprioception at 50 Hz minimum, time sync under 10 ms across all streams, a 60-task manipulation list with reset criteria, an annotation rubric, and delivery in LeRobot format with a schema doc, following the conventions the ecosystem inherited from Open X-Embodiment and the ALOHA line of work (arxiv.org/abs/2310.08864; arxiv.org/abs/2304.13705).
The immediate effect was that vendor quotes became comparable for the first time. The secondary effect was internal: the ML and data teams had to agree on what they were actually buying, which surfaced two silent disagreements about camera placement that had been corrupting cross-vendor consistency for months.
Step 2: RFP and Scorecard
The RFP went to six vendors with fifteen questions requiring numeric answers: committed usable-hour yield, sync tolerance and recheck cadence, audit accuracy, rig counts behind throughput claims, per-hour pricing by task complexity, pilot terms, and licensing. Four vendors answered with numbers. Two answered with adjectives and were dropped without further calls, which is the red flags screen doing its job cheaply.
The four remaining responses were scored independently by two people on the 10-criterion weighted scorecard (quality SLAs 15, modality coverage 12, calibration/sync 12, down to IP/licensing at 7). Independent scoring earned its keep in the reconciliation meeting: the two scorers disagreed by more than a point on only three criteria, and every disagreement traced to an ambiguous vendor answer rather than a difference in judgment. Those ambiguities went back to the vendors as written clarification requests, which is a politer and more useful outcome than one person’s optimism deciding the ranking.
The scoring separated the field more on contract terms than on hardware: two vendors wanted broad rights to reuse delivered episodes for other customers, which the team treated as disqualifying on the licensing criterion. Two finalists advanced.
Step 3: The 50-Hour Paid Pilots
Both finalists ran paid 50-hour pilots under production conditions, judged on four numbers fixed in the pilot agreements before capture began. This is the step that paper evaluation cannot replace, and the results diverged more than the scorecards had predicted.
| Pilot metric (agreed in advance) | Threshold | Finalist 1 | Finalist 2 |
|---|---|---|---|
| Usable-hour yield vs acceptance spec | ≥85% | 90% | 78% |
| Annotation accuracy (independent 5% re-label) | ≥97% | 98.2% | 96.1% |
| Policy success delta on fixed 12-task eval | Positive | +6 points vs baseline | +1 point |
| Loader time into LeRobot pipeline | ≤1 engineer-day | ~2 hours | Failed (schema mismatches, 3+ days) |
The policy-delta test deserves a note. The team fine-tuned the same fixed baseline policy twice, once on their existing mix, once with the 50 pilot hours added, and evaluated both on the same 12 tasks. Fifty hours is a small delta at VLA scale, and they treated it that way: not as proof of final performance, but as a canary. Data that helps at 50 hours might help at 1,000; data that does nothing at 50 hours is a warning you can act on before signing an annual contract.
Step 4: Contract and the Twelve Weeks After
The contract encoded the pilot numbers rather than replacing them with prose: yield commitment at 88 percent with redelivery of failures, sync tolerance and recalibration cadence as delivery requirements, LeRobot schema versioned in an appendix, quarterly re-audit rights, and no vendor reuse of delivered episodes.
The results over the first 1,000 contracted hours were unremarkable in the best sense. Yield held between 88 and 90 percent, within two points of the pilot. Engineering time per delivery fell from roughly a day to under an hour, because the format and schema were locked before signature instead of negotiated after each batch. The two silent disagreements about camera placement did not recur, because the spec, not tribal memory, was the reference.
The honest caveat: this process cost about five weeks end to end and roughly $7,000 in pilot fees across two finalists. For a team spending six figures annually on data, that insurance premium was around 5 percent of program cost. For a team buying 20 exploratory hours, it would be overkill, and we say so in the full playbook.
Frequently Asked Questions
How much did the procurement rebuild cost compared to what it saved?
About five weeks of process time and roughly $7,000 in paid pilot fees. Against it: yield moved from ~74 to ~90 percent, and per-delivery engineering time dropped from a day to under an hour, which on a six-figure annual program repaid the process cost within the first deliveries.
Why run paid pilots with two finalists instead of one?
Because scorecards rank paper, and paper can lie by omission. In this case the two finalists scored within a few points of each other, then diverged by 12 yield points and a failed loader test under production conditions.
Is a 50-hour pilot enough to judge data for a VLA model?
It is enough to judge the vendor’s pipeline, not the model’s ceiling. Treat the policy success delta as a canary: positive movement at 50 hours justifies scaling; zero movement is a cheap warning.
What belongs in the contract that most teams leave out?
The pilot numbers themselves: yield commitment with redelivery terms, sync tolerance as a delivery requirement, a versioned schema appendix, re-audit rights, and an explicit ban on vendor reuse of delivered episodes.
