Skip to main content

Dexset

dexset
Menu
Request a Demo

Tag: Vendor Evaluation

5 Hidden Costs in Robot Training Data Budgets (and How to Solve Each One)

Posted on by Sainath Gupta

Two teams we worked alongside last year signed for near-identical manipulation datasets at headline rates within a few dollars of each other. One closed its program roughly on budget. The other overran by nearly half and cut a planned training run to pay for it. The difference was never the rate. It was five quieter line items that neither quote itemized, and only one team went looking for them before signing.

I review DexSet’s QA ledgers, so I watch this pattern from the inside, and it supports one claim, which is the thesis of this post: data budgets die in the gap between quoted raw hours and delivered usable hours, and the five costs below are that gap, itemized. Teams negotiate hard on the dollars-per-hour figure, sign, and then watch these quieter items add 30 to 80 percent to the program. By the time the overrun is visible, the training run is scheduled and there is no negotiating position left.

These five costs stay hidden for a structural reason: they live in the gaps between quote and delivery. A quote prices raw hours; delivery is measured in usable, annotated, retrievable hours. Everything between those two definitions is where money leaks, and vendors have little incentive to itemize a gap that flatters their pricing.

This post names the five, puts our real numbers on each, and gives you the fix. All figures come from DexSet’s own pipeline benchmarks and program ledgers; where public hardware anchors exist, like the roughly $20k ALOHA rig, they are cited.

Read it before you sign anything. Then take the checklist at the bottom into your next vendor call.

Key Takeaways

  • QA rejection (10 to 30% in our pipelines) is the largest hidden cost: a $40 quote at 25% rejection is really $53.33 per usable hour.
  • Annotation scope creep adds $8 to $25 per hour per pass; label only selected data, not the whole corpus.
  • Rig downtime and recalibration silently cut utilization; every idle shift raises amortization per hour.
  • Storage, egress, and versioning run 3 to 6% of capture spend and spike at training time.
  • Operator turnover resets a 30 to 50% throughput learning curve; retention is a data-cost lever.

1. QA Rejection: The Gap Between Raw Hours and Usable Hours

QA rejection cost is the money spent capturing episodes that never enter your training set, and it is invisible in any quote expressed per raw hour. In our pipelines, 10 to 30 percent of raw episodes fail review: dropped frames, desynchronized views, occluded end-effectors, failed task completions.

The math bites harder than teams expect:

Quoted rate Rejection rate Real cost per usable hour Hidden premium
$30 10% $33.33 +11%
$40 25% $53.33 +33%
$50 30% $71.43 +43%

The fix. Three contract clauses. One: the vendor reports a measured rejection rate from a comparable program, not an aspiration. Two: recollection of rejected episodes is priced in writing, ideally on the vendor’s account above an agreed threshold. Three: you run a 100-to-200-hour paid pilot scored against your acceptance spec before any volume commitment. On our programs, a versioned task spec alone typically pulls rejection from the high 20s to under 15 percent within a few weeks.

2. Annotation Scope Creep: Paying for Labels You Never Train On

Annotation scope creep is the gradual expansion of labeling passes across an entire corpus when only a subset of the data needs them. Each pass costs real money: $8 to $12 per data hour for language instructions, $10 to $16 for subtask segmentation, $18 to $25 for dense masks and contact labels, per our benchmarks.

The failure mode is ordering “full annotation” on day one, before anyone knows which slices the model will actually consume. A 10,000-hour corpus with three blanket passes at a mid-range $38 per hour combined is $380,000 of labels, and in our experience a meaningful fraction of densely labeled episodes never influence a training run.

The fix. Stage it. Label language instructions broadly if your VLA needs them, then gate expensive passes behind data selection: annotate the episodes your curriculum actually samples. Run labeling in tranches with a two-week lag behind training experiments, so label spend follows demonstrated need. Teams that stage annotation typically spend 40 to 60 percent less on labels for the same eval performance.

3. Rig Downtime and Recalibration: Utilization Is the Denominator

Downtime cost is the amortization you pay while a rig is not collecting: maintenance, recalibration, resets, and idle shifts all raise the hardware cost of every hour that does get captured. A roughly $20k ALOHA-class cell (https://arxiv.org/abs/2304.13705), or a $32k Mobile ALOHA (https://arxiv.org/abs/2401.02117), is cheap only when it runs.

The numbers move fast. At two-shift utilization over 18 months, amortization is $4 to $9 per hour. Single shift with 50 percent idle time, and the same rig charges you $15 or more per hour before anyone touches a leader arm. Multi-camera exocentric arrays are worse offenders: calibration after every scene change eats 5 to 15 percent of scheduled collection time if it is not engineered out.

The fix. Treat utilization as a weekly KPI. Schedule calibration and scene resets into shift handovers, keep spare grippers and cameras on the shelf (a $600 spare beats a lost shift), and pre-stage scenes so operators walk into ready cells. If you are buying rather than building, ask the vendor how many shifts their rigs run; their utilization sets the amortization share baked into your rate.

4. Storage, Egress, and Versioning: The Bill That Arrives at Training Time

Data infrastructure cost is the spend on storing, versioning, and moving your dataset, and it stays invisible until the first big training run pulls the whole corpus out of cloud storage. Multi-view stereo capture generates terabytes per week; a 10,000-hour multi-camera program can produce several hundred terabytes before compression decisions are made.

Our planning figure is 3 to 6 percent of capture spend for storage, format conversion, and dataset versioning, with egress as the spike risk: pulling a few hundred terabytes across clouds at list egress prices can add tens of thousands of dollars per full-corpus read.

The fix. Decide storage format and residency before collection starts, not after. Co-locate data with training compute to kill egress. Standardize on a training-ready format on delivery (for example, LeRobot-compatible datasets, https://github.com/huggingface/lerobot, rather than raw ROS bags), so you pay conversion once. Version at the episode level so experiments pull slices, not the whole corpus.

5. Operator Turnover: The Learning Curve You Pay For Twice

Operator turnover cost is the throughput and quality you lose when a trained teleoperator leaves and a new one restarts the learning curve. In our programs, operators improve 30 to 50 percent in episodes-per-shift over their first 200 hours, and their rejection rates fall in parallel. Every departure resets both curves.

This cost hides inside blended rates. A vendor churning operators quietly delivers you a workforce that is permanently early-curve: slower, more rejected episodes, same invoice. You will never see a line item for it.

The fix. Ask vendors for operator tenure and how many hours their median operator has logged. In-house, pay experienced operators above generic labor rates; the throughput math justifies it easily. And instrument per-operator metrics, episodes per shift and rejection rate, so coaching happens before quality drifts.

The Pre-Signature Checklist

A pre-signature checklist converts these five hidden costs into questions a vendor must answer in writing before you commit volume:

  • ☐ Measured QA rejection rate on a comparable program, and who pays for recollection
  • ☐ Itemized rate card: capture, QA, each annotation pass ($8 to $25/hr range), infrastructure
  • ☐ Rig utilization (shifts per day) behind the amortization in the rate
  • ☐ Delivery format, storage residency, and who pays egress
  • ☐ Median operator tenure and hours logged
  • ☐ 100-to-200-hour paid pilot scored against your acceptance spec

If a vendor stalls on more than one of these, the hidden costs are not hidden from them. They are hidden from you. The full rate benchmarks behind every number in this post are published in our robot training data costs and pricing guide.

Audit Your Next Quote Against These Five

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the single biggest hidden cost in robot training data?

QA rejection. At the 10 to 30 percent rejection rates we measure, a quoted raw-hour rate understates the true cost per usable hour by 11 to 43 percent. It is the first number to demand from any vendor.

$8 to $25 per data hour per pass in our benchmarks: language instructions at the low end, dense masks and contact labels at the top. Staging annotation behind data selection typically cuts label spend 40 to 60 percent.

Plan 3 to 6 percent of capture spend, and engineer egress out by co-locating data with training compute. Multi-view stereo programs can reach hundreds of terabytes, so format and residency decisions belong before collection starts.

Throughput improves 30 to 50 percent over an operator’s first 200 hours and rejection falls in parallel. High-churn workforces deliver permanently early-curve performance at the same hourly rate.

Use the six-question checklist above: measured rejection rate, itemized rates, rig utilization, delivery format and egress liability, operator tenure, and a paid pilot against your spec.

Four Ways to Buy Robot Training Data, Compared: Pros, Cons, and Costs

Posted on by Sainath Gupta

Taped above a monitor in a buyer’s office we visited last year was a coffee-stained, three-page acceptance spec, its 10 ms sync tolerance circled twice in red pen. That battered document was doing more procurement work than the forty-page RFP folder on the shelf beside it, because it forced every vendor quote onto the same axes. It also marked its owners as unusual: teams rarely frame “how are we going to buy this” as a decision at all. They email two vendors someone met at CoRL, pick the cheaper quote, and only discover they chose a procurement approach when it fails.

The failure is predictable because each buying approach has a known cost structure and a known blind spot. An informal purchase is fast and blind. A full RFP is thorough and slow. A pilot-first approach measures what matters but covers one vendor at a time. Open datasets are free and almost never match your embodiment or task distribution.

This post compares the four approaches on speed, cost, and risk, with the math that lets you pick deliberately. It draws on the same scorecard and pilot protocol as our full Robotics Data Buyer’s Playbook, which is where the reusable templates live.

At DexSet we respond to all four buying styles weekly, so we see their outcomes from the supplier side: which approaches produce clean contracts and which produce disputes about what “an hour of data” was supposed to mean.

Key Takeaways

  • There are four common procurement approaches: informal purchase, full RFP + scorecard, pilot-first, and open-data-plus-top-up. Each has a distinct cost and risk profile.
  • Informal buying is cheapest to run ($0 process cost) and most expensive to survive: yield surprises routinely add 20 to 40 percent to effective cost.
  • The RFP + scorecard + pilot combination costs roughly 3 to 5 weeks and $1,500 to $4,500 in pilot fees, and is the only approach that measures quality before annual commitment.
  • Open datasets (Open X-Embodiment, DROID, Ego4D) are excellent for pretraining and mixing, but embodiment and task mismatch means most teams still purchase targeted data on top.

Approach 1: Informal Purchase

An informal purchase is vendor selection without a written specification, scoring method, or pilot: the buyer requests quotes, reviews sample clips, and signs with the most convincing option. It is how most first data purchases happen, and it is defensible exactly once, at very small volume, when you are still learning what to specify.

Pros: fastest path to first data (days, not weeks); no process overhead; fine for exploratory volumes under 20 hours.

Cons: samples are curated, so latent defects (sync offsets, calibration drift) go undetected; quotes are not comparable because no shared spec exists; no contractual yield commitment, so failed hours are your loss; format surprises arrive with the first delivery.

Cost profile: zero process cost up front. In our benchmarks, yield surprises and conversion work typically add 20 to 40 percent to effective per-usable-hour cost versus a piloted vendor. At 1,000 hours, that is $8,000 to $16,000 of avoidable spend on a $40/hr program.

Approach 2: Full RFP with Weighted Scorecard

A full RFP approach sends a written acceptance spec and a fixed question set to five to eight vendors, then scores responses on weighted criteria before any commitment. This is classic procurement discipline adapted to robot data: quality SLAs, calibration and sync specs, throughput evidence, pricing transparency, licensing terms.

Pros: quotes become comparable because everyone bids the same spec; weak vendors self-eliminate (in our experience, roughly a third of recipients answer with adjectives instead of numbers); the scorecard creates an audit trail for the decision; licensing and consent problems surface before signature.

Cons: takes two to four weeks; still paper-based, so a vendor can score well and underdeliver; overkill below roughly $25,000 in annual data spend.

Cost profile: the process costs internal time only, typically 20 to 30 person-hours across spec writing, scoring, and reconciliation. It buys you comparability and eliminates the worst outcomes, but on its own it does not measure production quality.

Approach 3: Pilot-First

A pilot-first approach skips broad solicitation and goes straight to a 50-hour paid pilot with one or two candidate vendors, judged on predefined metrics: usable-hour yield, annotation audit accuracy, policy success delta on a fixed eval set, and loader time into LeRobot or RLDS. It optimizes for measured evidence over paper promises.

Pros: measures the only thing that matters, production output; small, fixed downside ($1,500 to $4,500 per pilot at market rates); fast when you already know the credible vendors; the policy-delta test catches defects no document review can.

Cons: covers only the vendors you pilot, so a better option may never be evaluated; sequential pilots take longer than parallel paper scoring; requires you to have a stable eval task set and baseline policy, which very early teams may lack.

Cost profile: $3,000 to $9,000 to pilot two vendors, plus about one engineer-week for evaluation. Expensive compared to reading PDFs, cheap compared to one bad quarter of deliveries.

Approach 4: Open Data Plus Targeted Top-Up

The open-data approach builds the base training mix from public corpora, then purchases only the targeted data the public sets cannot provide. The public layer is genuinely strong now: Open X-Embodiment spans over one million episodes across 22 embodiments (arxiv.org/abs/2310.08864), DROID adds 76,000 diverse teleop episodes (arxiv.org/abs/2403.12945), and Ego4D provides thousands of hours of egocentric human video (arxiv.org/abs/2110.07058), most of it accessible through Hugging Face dataset cards and RLDS tooling.

Pros: near-zero acquisition cost for pretraining scale; well-documented formats (RLDS, LeRobot conversions); community-validated quality.

Cons: embodiment mismatch (your gripper, camera placement, and control rates differ from the source robots); task distribution rarely matches your product; licenses vary and some restrict commercial use, so legal review is not optional; fine-tuning still demands in-domain demonstrations, which puts you back in one of the first three approaches for the data that moves your metrics most.

Cost profile: storage and engineering only for the public layer, then standard market rates ($28 to $60 per teleop hour, $15 to $40 per egocentric hour in our benchmarks) for the top-up volume, which is typically 10 to 30 percent of total hours but drives most of the task-specific performance.

Side-by-Side Comparison

The four approaches differ most in where they spend money: process time up front, or rework after delivery.

Approach Time to contract Process cost Quality measured before commitment? Typical effective cost penalty vs piloted baseline Best for
Informal purchase 3–10 days ~$0 No +20–40% Exploratory buys under 20 hours
Full RFP + scorecard 2–4 weeks 20–30 person-hours Partially (paper only) +5–15% Annual spend above $25k, multiple candidate vendors
Pilot-first 2–3 weeks per vendor $1,500–$4,500 per pilot Yes Baseline Teams with stable eval tasks and known vendor shortlist
Open data + top-up 1–2 weeks (legal + integration) Engineering time Yes for public layer, no for top-up unless piloted Depends on top-up approach Pretraining scale plus targeted fine-tuning

The pattern most mature buyers converge on is a hybrid: RFP to filter the field, scorecard to rank it, pilot to verify the winner, open data underneath it all as the pretraining base. That sequence is exactly what the Robotics Data Buyer’s Playbook packages, including the scorecard weights and pilot pass/fail thresholds.

One sequencing note from the supplier side: run the approaches in that order, not in parallel. Teams that pilot before writing a spec end up measuring vendors against criteria invented after the data arrived, which makes the results unarguable in exactly the wrong way; nobody can agree what a pass looks like. Teams that RFP without a spec get six incomparable quotes and mistake the spread for market variance. The spec is upstream of everything, takes about a week to write, and is the only artifact in the process that costs nothing but attention.

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

Which robot data procurement approach is cheapest overall?

The RFP-plus-pilot hybrid, once volume passes roughly $25,000 per year. Informal buying has the lowest process cost but the highest effective cost, because yield surprises add 20 to 40 percent on typical programs.

Not fully. Open X-Embodiment, DROID, and Ego4D are strong pretraining bases, but embodiment and task mismatch means fine-tuning still needs in-domain demonstrations, typically 10 to 30 percent of total hours purchased to spec.

For exploratory volumes under about 20 hours, where the goal is learning what to specify rather than feeding a production training run. Anything feeding a release model deserves at least a pilot.

Send the RFP to five to eight vendors, score all responses, then pilot the top one or two. Piloting more than two rarely changes the decision and doubles the evaluation load.

Four numbers agreed in advance: usable-hour yield (target 85 percent or higher), annotation accuracy on an independently re-labeled 5 percent sample (97 percent or higher), policy success delta on a fixed eval set, and loader time into your training format (one engineer-day or less).

Why Data Procurement, Not Data Capture, Is the Biggest Bottleneck in Physical AI

Posted on by Enoch Pakanati

Two teams approached us in the same quarter with nearly identical VLA data programs. The first sent a two-page acceptance spec and an RFP that demanded numbers, then ran a paid pilot before signing with anyone. The second compared three price sheets and took the lowest. Half a year later, the first team’s deliveries were entering training runs the day they arrived; the second team was still bisecting a training regression that traced back to a sync tolerance no contract had ever specified. Same budget class, same architecture, divergent quarters. The difference was not capture quality. It was procurement.

The uncomfortable part is that capture itself has scaled. ALOHA-class rigs are reproducible from public documentation (arxiv.org/abs/2304.13705). Open X-Embodiment pooled over a million episodes across 22 embodiments (arxiv.org/abs/2310.08864). DROID collected 76,000 teleop episodes across 13 institutions (arxiv.org/abs/2403.12945). The hardware and process knowledge exist. What has not scaled is the buying side: most teams still purchase demonstration data with less rigor than they apply to a laptop refresh.

This post makes the case that procurement is now the binding constraint, shows what the gap costs in numbers, and gives you the four artifacts that close it. It condenses the full Robotics Data Buyer’s Playbook, which includes the complete scorecard and RFP question bank.

We see this from the vendor side at DexSet. The buyers who send a real spec get better data at better prices than the buyers who send a budget and a hope, because a real spec lets us commit to numbers instead of hedging against unknowns.

Key Takeaways

  • Capture capacity has commoditized; vendor selection has not. Weak procurement is now the most common cause of stalled VLA data programs.
  • The cost of a bad pick is measured in usable hours: a cheap vendor at 70 percent yield can cost more per usable hour than a pricier one at 90 percent, once rework is priced in.
  • Latent defects (sync error, calibration drift) are invisible in previews and only surface in training, which is why sample-based buying fails.
  • The fix is procedural, not heroic: a spec-first RFP, a 10-criterion weighted scorecard, a red flags list, and a 50-hour paid pilot.

What Makes Procurement the Bottleneck

The procurement bottleneck is the delay and waste created when robot data purchasing decisions are made without a specification, a scoring method, or a pilot, forcing quality problems to surface downstream during training. It shows up as three concrete failure patterns.

Pattern one: the invisible defect. Robot data defects are latent. A 40 ms sync offset between camera frames and joint states will not appear in video playback, but it corrupts the state-action mapping your imitation learning policy depends on. Teams that buy on sample previews systematically miss this class of problem, then spend weeks bisecting training regressions that were purchased, not coded.

Pattern two: the spec vacuum. When the buyer has no written acceptance spec, every vendor quote describes a different product. One vendor’s “hour of manipulation data” is stereo egocentric plus wrist cameras with dense annotations; another’s is a single 720p exocentric mono stream. Comparing their prices is meaningless, and procurement stalls in clarification loops that a two-page spec would have prevented.

Pattern three: the format tax. Deliveries arrive in whatever the vendor uses internally: undocumented HDF5, half-converted rosbag2, a proprietary container. Your engineers pay the conversion cost into LeRobot or RLDS on every delivery. In our experience that tax runs 15 to 25 percent of program cost when format compatibility was never contracted.

The Cost of Buying Badly, in Numbers

The cost of weak procurement is best expressed as price per usable hour, which is the delivered price divided by the fraction of hours that pass your acceptance spec. The sticker price is the number vendors compete on; the usable-hour price is the number your training run experiences.

Scenario Sticker price Usable-hour yield True price per usable hour 1,000-hour program cost (usable basis)
Cheapest bid, no pilot $32/hr 70% $45.71 $45,710
Mid bid, sample-only check $40/hr 82% $48.78 $48,780
Higher bid, passed 50-hr pilot $46/hr 90% $51.11 $51,110
Cheapest bid after rework and triage time $32/hr + engineering time 70% $55–$60 effective $55,000–$60,000

Read the last row carefully. The cheapest vendor is the most expensive one once you price the engineering time spent triaging failures and patching training mixes, and that is before counting the schedule slip, which no spreadsheet captures but every roadmap feels. These are typical ranges from our benchmarks; your yields will vary, which is exactly why you measure them in a pilot instead of assuming them.

The Fix: Four Artifacts, Not More Meetings

The fix for the procurement bottleneck is a set of four reusable artifacts that convert vendor selection from judgment calls into measurements. Each one is small. Together they remove the guesswork that creates the bottleneck.

  • A spec-first RFP. Write the acceptance spec (cameras, sync tolerance under 10 ms, task list, annotation rubric, delivery format) before contacting anyone. Then send 15 questions that force numeric answers: committed usable-hour yield, audit accuracy, rig counts, per-hour pricing by task complexity.
  • A weighted scorecard. Ten criteria, weights summing to 100, with data quality SLAs, modality coverage, and calibration/sync spec carrying 39 points between them. Two independent scorers, reconciled. Automatic disqualification for weak answers on quality, calibration, or licensing.
  • A red flags list. Some behaviors end the conversation regardless of score: pricing only after discovery calls, refusal of paid pilots, proprietary-only formats, no consent documentation, broad data reuse rights.
  • A 50-hour paid pilot. Production rigs, production operators, and four predefined numbers: usable-hour yield (85 percent or higher), annotation audit accuracy (97 percent or higher on a 5 percent sample), policy success delta on a fixed eval set, and loader time into your LeRobot or RLDS pipeline (one engineer-day or less).

The full versions, including the complete scorecard weights and all 15 RFP questions, are in the Robotics Data Buyer’s Playbook.

How to Tell If Procurement Is Your Bottleneck

A procurement bottleneck diagnosis is a check of where data-program time actually goes, and it takes one honest hour with your delivery logs. Run through five questions:

  • Can anyone state your acceptance spec from memory, or point to the document? If the spec lives in tribal knowledge, every vendor conversation is renegotiating it implicitly.
  • What fraction of delivered hours reached a training run last quarter? If nobody tracks this number, assume it is worse than you think; teams that start measuring usually find 15 to 30 percent of purchased hours never trained anything.
  • How long does a delivery take to enter the pipeline? More than a few hours of engineer time per batch means you are paying the format tax on every delivery.
  • Could you defend your current vendor choice to your board with numbers? A scorecard produces that defense as a byproduct. A demo-based decision cannot.
  • When did a data defect last cost you a training debugging cycle? If the answer is “this quarter,” the defect entered through procurement, not through capture.

Two or more uncomfortable answers means the bottleneck is upstream of your training code, and the fix below is cheaper than the symptom.

Scaling the Playbook Across Programs

Scaling this process means running the same artifacts on every purchase rather than reinventing evaluation per deal. The spec becomes a living document versioned alongside your model releases. The scorecard weights shift as your risks shift: teams early in data collection weight modality coverage higher; teams scaling a proven recipe weight throughput and SLAs higher. Pilot results accumulate into an internal vendor database, which is the closest thing this industry has to a track record. After three or four cycles, vendor evaluation drops from weeks of meetings to days of scoring, and that is the bottleneck removed.

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

Why is procurement a bigger bottleneck than capture in physical AI?

Because capture methods are now well documented (ALOHA, DROID, Open X-Embodiment) while buying practices are not. Latent defects like sync error survive sample review and surface in training, so unstructured purchasing converts directly into stalled training runs.

Delivered price divided by the fraction of hours passing your acceptance spec. A $32/hr vendor at 70 percent yield costs $45.71 per usable hour, which is why sticker-price comparisons mislead.

Around 50 paid hours. That volume exposes process problems (calibration drift, operator variance, QA gaps) while keeping a failed pilot cheap: typically $1,500 to $4,500 at market rates.

Native delivery in LeRobot, RLDS/TFDS, or documented HDF5, with MCAP or rosbag2 as ROS 2 options. Proprietary-only delivery is a disqualifier because it adds conversion cost to every delivery and blocks independent audits.