Taped above a monitor in a buyer’s office we visited last year was a coffee-stained, three-page acceptance spec, its 10 ms sync tolerance circled twice in red pen. That battered document was doing more procurement work than the forty-page RFP folder on the shelf beside it, because it forced every vendor quote onto the same axes. It also marked its owners as unusual: teams rarely frame “how are we going to buy this” as a decision at all. They email two vendors someone met at CoRL, pick the cheaper quote, and only discover they chose a procurement approach when it fails.
The failure is predictable because each buying approach has a known cost structure and a known blind spot. An informal purchase is fast and blind. A full RFP is thorough and slow. A pilot-first approach measures what matters but covers one vendor at a time. Open datasets are free and almost never match your embodiment or task distribution.
This post compares the four approaches on speed, cost, and risk, with the math that lets you pick deliberately. It draws on the same scorecard and pilot protocol as our full Robotics Data Buyer’s Playbook, which is where the reusable templates live.
At DexSet we respond to all four buying styles weekly, so we see their outcomes from the supplier side: which approaches produce clean contracts and which produce disputes about what “an hour of data” was supposed to mean.
Key Takeaways
- There are four common procurement approaches: informal purchase, full RFP + scorecard, pilot-first, and open-data-plus-top-up. Each has a distinct cost and risk profile.
- Informal buying is cheapest to run ($0 process cost) and most expensive to survive: yield surprises routinely add 20 to 40 percent to effective cost.
- The RFP + scorecard + pilot combination costs roughly 3 to 5 weeks and $1,500 to $4,500 in pilot fees, and is the only approach that measures quality before annual commitment.
- Open datasets (Open X-Embodiment, DROID, Ego4D) are excellent for pretraining and mixing, but embodiment and task mismatch means most teams still purchase targeted data on top.
Approach 1: Informal Purchase
An informal purchase is vendor selection without a written specification, scoring method, or pilot: the buyer requests quotes, reviews sample clips, and signs with the most convincing option. It is how most first data purchases happen, and it is defensible exactly once, at very small volume, when you are still learning what to specify.
Pros: fastest path to first data (days, not weeks); no process overhead; fine for exploratory volumes under 20 hours.
Cons: samples are curated, so latent defects (sync offsets, calibration drift) go undetected; quotes are not comparable because no shared spec exists; no contractual yield commitment, so failed hours are your loss; format surprises arrive with the first delivery.
Cost profile: zero process cost up front. In our benchmarks, yield surprises and conversion work typically add 20 to 40 percent to effective per-usable-hour cost versus a piloted vendor. At 1,000 hours, that is $8,000 to $16,000 of avoidable spend on a $40/hr program.
Approach 2: Full RFP with Weighted Scorecard
A full RFP approach sends a written acceptance spec and a fixed question set to five to eight vendors, then scores responses on weighted criteria before any commitment. This is classic procurement discipline adapted to robot data: quality SLAs, calibration and sync specs, throughput evidence, pricing transparency, licensing terms.
Pros: quotes become comparable because everyone bids the same spec; weak vendors self-eliminate (in our experience, roughly a third of recipients answer with adjectives instead of numbers); the scorecard creates an audit trail for the decision; licensing and consent problems surface before signature.
Cons: takes two to four weeks; still paper-based, so a vendor can score well and underdeliver; overkill below roughly $25,000 in annual data spend.
Cost profile: the process costs internal time only, typically 20 to 30 person-hours across spec writing, scoring, and reconciliation. It buys you comparability and eliminates the worst outcomes, but on its own it does not measure production quality.
Approach 3: Pilot-First
A pilot-first approach skips broad solicitation and goes straight to a 50-hour paid pilot with one or two candidate vendors, judged on predefined metrics: usable-hour yield, annotation audit accuracy, policy success delta on a fixed eval set, and loader time into LeRobot or RLDS. It optimizes for measured evidence over paper promises.
Pros: measures the only thing that matters, production output; small, fixed downside ($1,500 to $4,500 per pilot at market rates); fast when you already know the credible vendors; the policy-delta test catches defects no document review can.
Cons: covers only the vendors you pilot, so a better option may never be evaluated; sequential pilots take longer than parallel paper scoring; requires you to have a stable eval task set and baseline policy, which very early teams may lack.
Cost profile: $3,000 to $9,000 to pilot two vendors, plus about one engineer-week for evaluation. Expensive compared to reading PDFs, cheap compared to one bad quarter of deliveries.
Approach 4: Open Data Plus Targeted Top-Up
The open-data approach builds the base training mix from public corpora, then purchases only the targeted data the public sets cannot provide. The public layer is genuinely strong now: Open X-Embodiment spans over one million episodes across 22 embodiments (arxiv.org/abs/2310.08864), DROID adds 76,000 diverse teleop episodes (arxiv.org/abs/2403.12945), and Ego4D provides thousands of hours of egocentric human video (arxiv.org/abs/2110.07058), most of it accessible through Hugging Face dataset cards and RLDS tooling.
Pros: near-zero acquisition cost for pretraining scale; well-documented formats (RLDS, LeRobot conversions); community-validated quality.
Cons: embodiment mismatch (your gripper, camera placement, and control rates differ from the source robots); task distribution rarely matches your product; licenses vary and some restrict commercial use, so legal review is not optional; fine-tuning still demands in-domain demonstrations, which puts you back in one of the first three approaches for the data that moves your metrics most.
Cost profile: storage and engineering only for the public layer, then standard market rates ($28 to $60 per teleop hour, $15 to $40 per egocentric hour in our benchmarks) for the top-up volume, which is typically 10 to 30 percent of total hours but drives most of the task-specific performance.
Side-by-Side Comparison
The four approaches differ most in where they spend money: process time up front, or rework after delivery.
| Approach | Time to contract | Process cost | Quality measured before commitment? | Typical effective cost penalty vs piloted baseline | Best for |
|---|---|---|---|---|---|
| Informal purchase | 3–10 days | ~$0 | No | +20–40% | Exploratory buys under 20 hours |
| Full RFP + scorecard | 2–4 weeks | 20–30 person-hours | Partially (paper only) | +5–15% | Annual spend above $25k, multiple candidate vendors |
| Pilot-first | 2–3 weeks per vendor | $1,500–$4,500 per pilot | Yes | Baseline | Teams with stable eval tasks and known vendor shortlist |
| Open data + top-up | 1–2 weeks (legal + integration) | Engineering time | Yes for public layer, no for top-up unless piloted | Depends on top-up approach | Pretraining scale plus targeted fine-tuning |
The pattern most mature buyers converge on is a hybrid: RFP to filter the field, scorecard to rank it, pilot to verify the winner, open data underneath it all as the pretraining base. That sequence is exactly what the Robotics Data Buyer’s Playbook packages, including the scorecard weights and pilot pass/fail thresholds.
One sequencing note from the supplier side: run the approaches in that order, not in parallel. Teams that pilot before writing a spec end up measuring vendors against criteria invented after the data arrived, which makes the results unarguable in exactly the wrong way; nobody can agree what a pass looks like. Teams that RFP without a spec get six incomparable quotes and mistake the spread for market variance. The spec is upstream of everything, takes about a week to write, and is the only artifact in the process that costs nothing but attention.
Frequently Asked Questions
Which robot data procurement approach is cheapest overall?
The RFP-plus-pilot hybrid, once volume passes roughly $25,000 per year. Informal buying has the lowest process cost but the highest effective cost, because yield surprises add 20 to 40 percent on typical programs.
Can open datasets replace purchased robot data?
Not fully. Open X-Embodiment, DROID, and Ego4D are strong pretraining bases, but embodiment and task mismatch means fine-tuning still needs in-domain demonstrations, typically 10 to 30 percent of total hours purchased to spec.
When is an informal purchase acceptable?
For exploratory volumes under about 20 hours, where the goal is learning what to specify rather than feeding a production training run. Anything feeding a release model deserves at least a pilot.
How many vendors should get the RFP versus the pilot?
Send the RFP to five to eight vendors, score all responses, then pilot the top one or two. Piloting more than two rarely changes the decision and doubles the evaluation load.
What metrics should a pilot measure regardless of approach?
Four numbers agreed in advance: usable-hour yield (target 85 percent or higher), annotation accuracy on an independently re-labeled 5 percent sample (97 percent or higher), policy success delta on a fixed eval set, and loader time into your training format (one engineer-day or less).