Skip to main content

Dexset

dexset
Menu
Request a Demo

Tag: Best Practices

5 Hidden Challenges in Training Data for Vision-Language-Action (VLA) Models and How to Solve Them

Posted on by Sainath Gupta

Spot-checking your robot dataset is making things worse. Not the data, your judgment of it: per-episode visual review manufactures confidence, while the defects that actually sink VLA fine-tunes live at the distribution level, where no individual episode looks wrong. That is the thesis of this post, and its practical consequence: the unit of QA for VLA data is the batch and the corpus, not the episode.

The obvious VLA training data quality problems get solved early because they are loud. Not enough episodes, missing camera streams, no language labels: these announce themselves in week one. The problems that actually sink fine-tunes are quiet. They pass a visual spot-check, survive ingestion without errors, and only surface weeks later as a loss curve that plateaus early or a policy that fumbles tasks it should own.

These failures stay hidden for a structural reason. A robot dataset has no compiler. Nothing throws an exception when your timestamps drift 40 ms or when 92% of your instructions share one sentence shape. The data trains. The model just trains worse, and the gap gets attributed to architecture, hyperparameters, or “needing more data,” which leads teams to buy more of the same defective distribution.

We gate every dataset that leaves DexSet through a QA pipeline built around these exact failure modes, so this post is written from the reject pile. For each of the five challenges: what it is, how it hides, how it damages training, and the check that catches it. All five checks are runnable before you spend a single GPU-hour.

Key Takeaways – The five quiet VLA training data quality failures: frame-action timestamp skew, instruction monoculture, silent failure episodes, embodiment/config drift, and mixture starvation. – Each one passes casual inspection and each one measurably degrades fine-tunes; contamination rates as low as 5-10% flatten gains on precision tasks in our ablations. – Every one has a cheap automated check; together they cost $5-9 per data-hour in our pipeline, a fraction of the GPU spend they protect. – Diversity metrics belong in acceptance criteria, not in hopes: template share, verb counts, and per-task hour caps should be contractual.

Challenge 1: Frame-Action Timestamp Skew

Timestamp skew is a systematic offset or drift between when a camera frame was captured and when the recorded action actually executed. Software timestamps on separate device clocks drift, USB cameras buffer unpredictably, and a rig that was aligned in January is often 30-50 ms off by March.

Why it hides: video looks fine, actions replay fine, and nothing is corrupt. Why it hurts: modern VLAs predict action chunks, and continuous-action heads (diffusion in Octo, arxiv.org/abs/2405.12213; flow matching in pi-zero, arxiv.org/abs/2410.24164) regress targets whose alignment with observations is the whole training signal. Skew smears that mapping. In our fine-tune ablations, loss curves separate measurably once skew passes about 10 ms, which is where we set our rig tolerance.

The fix: hardware timestamping where possible, a physical sync event at episode start (a visible gripper snap works), and an automated cross-correlation check between commanded motion and observed motion per episode. Reject on drift, do not average it away.

Challenge 2: Instruction Monoculture

Instruction monoculture is a label distribution where most episodes share a handful of sentence templates, regardless of how many total instructions exist. A 100,000-episode corpus with 14 templates is monoculture at scale; we audited exactly that situation in a recent humanoid client engagement.

Why it hides: ataset cards report episode counts, not instruction diversity – the metric that actually predicts language generalization. Why it hurts: the model learns the templates as fixed keys rather than learning language. RT-1-era templated labels produced policies with a vocabulary of about a dozen verbs (arxiv.org/abs/2212.06817); the semantic breadth in RT-2 came from web co-training, not from the robot labels (arxiv.org/abs/2307.15818). If your labels are monoculture, paraphrase failure is guaranteed.

The fix: measure template share, unique verbs, and spatial-relation coverage per batch, and set hard thresholds. We cap any single template at 8% of a batch and run paraphrase augmentation on failures. Priced honestly, a proper natural-language annotation pass adds $8-15 per data-hour. It is the least skippable line in the budget.

Challenge 3: Silent Failure Episodes

A silent failure episode is a demonstration where the task did not actually succeed – the most damaging robot dataset QA gap in teleoperation-based collection but the episode carries no failure label, so it trains the policy on a wrong outcome presented as a right one. teleoperators drop objects, miss insertions by a few millimeters, or complete the wrong variant – which is why teleoperation data collection requires explicit success criteria baked into the session protocol of an ambiguous task, and at collection pace nobody logs it.

Why it hides: failures look like successes at thumbnail resolution, and per-episode review feels too expensive so teams sample 2% and hope. Why it hurts: imitation learning has no mechanism to discount bad demonstrations; a 5-10% contamination rate is enough to visibly flatten fine-tune gains on precision tasks in our ablations.

The fix: explicit success criteria written per task family before collection, a success label as a required field, and dual review on a stratified sample weighted toward long-horizon and precision tasks. This is most of what our $5-9 per data-hour teleoperation data quality review buys.

Challenge 4: Embodiment and Configuration Drift

Configuration drift is unannounced variation in the physical setup across a collection: cameras nudged between sessions, gripper fingers swapped after wear, firmware updates that change controller latency, tables raised two centimeters. Each change is small. Accumulated, your “single embodiment” corpus is quietly five embodiments.

Why it hides: no single episode is wrong, so episode-level QA passes everything. Why it hurts: the policy either averages across configurations it cannot distinguish or keys on incidental visual cues to tell them apart. Cross-embodiment training works when variation is labeled, which is exactly how Open X-Embodiment makes 22 embodiments useful (arxiv.org/abs/2310.08864); unlabeled variation is just noise.

The fix: a per-session calibration ritual (camera extrinsics check, gripper state, controller version logged into episode metadata) and drift detection that compares session-level statistics. Metadata is free at collection time and unrecoverable afterward.

Challenge 5: Mixture Starvation

Mixture starvation is a training mixture where one data source or task family dominates sampling, starving the components that were the reason you built the mixture. It happens by accident: datasets differ in episode length, so uniform episode sampling silently overweights short-episode sources.

Why it hides: the mixture config looks balanced on paper while the effective batch composition is not. Why it hurts: OpenVLA and Octo both hand-tuned per-dataset mixture weights rather than sampling naively, and OpenVLA’s curated 970k-episode mixture is a large part of how a 7B model beat a 55B one (arxiv.org/abs/2406.09246). Teams that skip mixture sampling accounting also routinely drop web co-training too low and watch open-vocabulary grounding decay.

The fix: account mixtures in timesteps or tokens, not episodes; log effective composition per training run; cap per-task-family hours at collection time so the raw corpus starts balanced (we use a 60-hour cap per family in typical fine-tuning collections).

A note on why these five cluster together: they are all distribution problems rather than instance problems. Any single episode can pass inspection while the corpus as a whole is defective, which is why per-episode spot checks feel reassuring and prove nothing. The unit of QA for VLA data is the batch and the corpus. Once a team internalizes that, the checks below stop looking like overhead and start looking like the cheapest model improvement available.

How Do You Check VLA Training Data Quality? (5 Checks)

ChallengeCheap Detection CheckThreshold We Use
Timestamp skewCommand-vs-observed motion cross-correlation per episodeReject > 10 ms
Instruction monocultureTemplate share + unique-verb count per 1,000-episode batchNo template > 8%
Silent failuresSuccess criteria + stratified dual review5% dual-review floor
Configuration driftSession metadata + session-statistics comparisonAny unlogged change fails
Mixture starvationTimestep-level composition logging per runMatch declared weights ±5%

Run these against any dataset, ours included, before training on it. The full framework, including costs and the vendor RFP scorecard these checks feed into, is in the complete guide to VLA training data.

Next Step

download our free LeRobot-format sample pack and run these five checks against it yourself; the QA report ships alongside the episodes. Or start with the complete VLA training data guide.

Request a Quote

Frequently Asked Questions

What are the most common quality problems in VLA training data?

The quiet ones: frame-action timestamp skew, instruction template monoculture, unlabeled failure episodes, unlogged configuration drift, and unbalanced training mixtures. All five pass casual inspection and all five measurably degrade fine-tunes.

For continuous-action models we hold frame-to-action skew under 10 ms, because our fine-tune ablations show measurable degradation beyond that. Discrete-token models tolerate more, but skew is cheap to control at capture time and expensive to fix later.

Measure instruction template share and unique-verb counts per batch, verify success labels on a stratified dual-reviewed sample, and cross-correlate commanded against observed motion for sync. All three checks run in hours on commodity hardware.

Usually because the new data repeats the old distribution: same tasks, same templates, same scene. Past roughly 50-150 demonstrations per task, marginal episodes add little; audit diversity before adding volume. This is the most common symptom of undetected VLA training data quality problems in the distribution.

5 Hidden Challenges in Data Capture Hardware for Embodied AI and How to Solve Them

Posted on by Sainath Gupta

Printing your timestamps is the checklist item almost every capture team skips, and skipping it is how a ten-thousand-episode dataset dies without a symptom. We watched it happen. The dataset looked perfect: clean video, complete joint logs, delivered on time. Then the client’s policy trained to a success rate barely above random, and three weeks of forensics traced it to a 60 ms clock offset between the wrist camera and the robot controller. Every action label in the dataset was stale by two frames, and a five-line timestamp check would have caught it before the first shift ended.

Failures like this stay hidden because capture hardware fails silently, and that is the argument this post makes: silent failure is the defining property of this hardware category, so the five failure modes below must be designed out before capture, not caught after it. A camera that stops recording is obvious. A camera whose clock drifts, whose calibration decays, or whose operator quietly degrades over an eight-hour shift produces data that looks fine and trains badly. Spec sheets describe none of this, because spec sheets describe devices, not fleets running sixty-hour weeks.

This post catalogs the five failure modes my QA team catches most often, what each one costs, and the specific fix for each. All five are hardware-level problems you can design out before your first episode is recorded; the device-by-device background lives in our full guide to data capture hardware for embodied AI.

I lead data quality at DexSet, where these checks run against every egocentric, teleoperation, and multi-view batch we deliver. The thresholds below are the ones on our own QA gates.

Key Takeaways – The five hidden challenges are clock drift, calibration decay, operator fatigue, offload bandwidth, and data format lock-in. None appear on a spec sheet. – Clock drift is the most expensive: tens of milliseconds of offset silently corrupts every action label in an episode. Our QA gate rejects episodes beyond 25 ms of cross-stream misalignment. – Calibration decays with heat and handling; we re-verify extrinsics on a fixed cadence (per shift for moved rigs) rather than trusting installation-day values. – Each challenge has a cheap hardware-stage fix and an expensive post-hoc one. The table below prices both.

Challenge 1: Clock Drift and Cross-Sensor Sync

Clock drift is the gradual divergence of independent device clocks, and it corrupts the observation-action alignment that imitation learning depends on. Consumer devices drift by seconds per day; even good oscillators disagree by milliseconds per hour. A dataset can pass every visual inspection while every frame’s action label describes what the robot did 60 ms later.

The fix is architectural. Prefer hardware with a native sync story: RealSense sync pins for camera arrays, Project Aria’s shared-clock VRS recordings for wearables, robot-side timestamping for teleop streams. Where hardware sync is impossible, inject a common observable event (a strobe, an audible clap, a scripted gripper snap) at episode boundaries and align in post. Our gate rejects any episode whose residual cross-stream misalignment exceeds 25 ms.

Challenge 2: Calibration Decay

Calibration decay is the slow invalidation of a rig’s intrinsic and extrinsic parameters through heat cycles, vibration, and handling. The extrinsics you measured at installation describe a rig that no longer exists three weeks later, once a camera mount has been bumped twice and a headset has been dropped once. Depth reprojection then smears by centimeters, which is fatal for grasp-point supervision.

The fix is cadence, not heroics. We re-verify extrinsics per shift on any rig that moves (headsets, handheld UMI-style grippers) and weekly on fixed studios, using a 60-second checkerboard or AprilTag routine baked into shift startup. Rigs that fail re-verification capture nothing until they pass. The routine costs about one percent of shift time and removes our second-largest historical rejection cause.

Challenge 3: Operator Fatigue as a Data Quality Variable

Operator fatigue is the degradation of demonstration quality over a shift, and it is a hardware property as much as a human one. A 600-plus-gram headset with an external battery pack produces different data at hour six than at hour one: slower corrections, shakier trajectories, more aborted grasps. The episodes still pass automated checks. The policy trained on them inherits the sloppiness.

The fix starts at procurement. Weigh the device; anything worn for full shifts should sit near action-camera weight (~150 g) rather than headset weight (500-650 g). For teleop, seated leader-arm designs in the ALOHA style (arxiv.org/abs/2304.13705) outlast free-space VR control on repetitive tasks. Then instrument for it: we track per-operator trajectory smoothness by shift-hour, and rotate stations when the curve bends.

Challenge 4: Offload Bandwidth and Storage Starvation

Offload bandwidth is the rate at which captured data moves off devices into storage, and when it is undersized the capture floor throttles itself. A six-camera RGB-D studio generates on the order of a terabyte per shift. Teams that budget for cameras but not for the offload path end a shift early to copy files, every day, forever, and consumer devices with slow card readers make it worse.

The fix is to size the pipe with the fleet: NVMe-based ingest per capture cell, hot-swappable media so devices never wait for uploads, and compression decisions made deliberately (lossless for depth and proprioception, visually lossless for RGB) rather than by device default. As a rule we budget storage-and-offload spend at 30 to 50 percent of camera spend; audits of stalled floors usually find it under 10 percent.

Challenge 5: Data Format Lock-In

Data format lock-in is dependence on a device’s proprietary recording format, and it converts every future pipeline change into a migration project. GoPro telemetry lives in GPMF tracks, ZED records SVO files, Aria writes VRS, every teleop stack invents an HDF5 layout, and none of them agree. The cost appears later, as engineer-weeks of converters and silent unit mismatches.

The fix is to normalize at ingest, on the same day the data is captured. Pick one canonical episode schema, the LeRobot dataset format (github.com/huggingface/lerobot) is the emerging community default, and treat device-native files as transient. Keep raw originals archived, but never let a training pipeline read a vendor format directly.

How to Audit Your Own Floor This Week

A capture audit is a one-week measurement pass that scores an existing pipeline against these five failure modes, and it needs no new hardware to run. The version we use internally:

  • Day 1: Print timestamps. Sample 50 random episodes and compute the median camera-to-proprioception offset per episode. Anything past 25 ms goes on the findings list.
  • Day 2: Re-run calibration on every rig and record the delta from stored values. Deltas above your reprojection tolerance mean decay is already in your data.
  • Day 3: Plot per-operator trajectory smoothness by shift-hour. A visible bend after hour four is a hardware ergonomics finding, not a staffing one.
  • Day 4: Time one full offload cycle per cell. Any cell where offload competes with capture time loses roughly a data hour per day.
  • Day 5: Count the distinct file formats between device and training job. More than one canonical format plus archived originals means converter debt is accruing.

Most teams find at least three of the five on the first pass. That is not an indictment; it is the baseline this hardware category imposes on everyone.

The Five Challenges at a Glance

Challenge Silent Symptom Cost If Ignored Fix at Hardware Stage Fix After the Fact
Clock drift Policies underperform despite clean-looking data Entire batches rejected or retrained Sync pins, shared clocks, boundary strobes (~$0-200/rig) Manual event alignment; often unrecoverable
Calibration decay Depth smears, grasp labels off by centimeters 10-20% rejection creep Per-shift AprilTag re-verification (1% of shift time) Re-calibration + partial batch salvage
Operator fatigue Quality slides after hour 4 Sloppy trajectories baked into the policy Buy light (~150 g worn), seated leader arms, rotation Filter late-shift episodes, lose volume
Offload bandwidth Shifts end early to copy files ~1 lost data hour/day per cell NVMe ingest, hot-swap media (30-50% of camera spend) Retrofit storage, disrupt operations
Format lock-in Converter scripts multiply Engineer-weeks per pipeline change Normalize to one schema at ingest Migration project across archived data

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the most common hidden failure in robot data capture hardware?

Cross-sensor clock drift. Independent device clocks diverge by milliseconds per hour, silently misaligning observations and action labels. It is the failure mode least visible in review and most damaging in training.

Our QA gates reject episodes with more than 25 ms of residual cross-stream misalignment for manipulation data. At 50-60 ms, action labels lag observations by one to two frames at 30 fps, which measurably degrades imitation learning.

Per shift for any rig that moves or is worn (headsets, handheld grippers), and weekly for fixed camera studios. A 60-second AprilTag verification at shift start catches decay before it contaminates a batch.

Yes, measurably. On our floors, trajectory smoothness and correction latency degrade in the back half of shifts on heavy worn devices, which is why device weight and seated leader-arm ergonomics are QA concerns, not comfort perks.

Normalize everything to one canonical episode schema at ingest; the LeRobot dataset format is the strongest community default in 2026. Archive device-native files (GPMF, SVO, VRS) but never train from them directly.

Every DexSet delivery passes the five gates above before it ships. [Download Sample Data] with the QA report attached, or start with the full guide to data capture hardware for embodied AI.

5 Hidden Challenges in Teleoperation Data Collection (and How to Solve Them)

Posted on by Sainath Gupta

One house rule on our collection floor reliably surprises visitors: no episode ships the day it is collected, however clean it looks on the monitor, because every episode waits for the overnight QA gate first. The rule exists because the visible challenges in teleoperation, buying rigs, hiring operators, picking a data format, are the ones teams plan for. The hidden challenges surface at episode 5,000, when the dataset is half-collected, the deadline is fixed, and the training run just exposed a flaw that has been silently poisoning every episode since week one.

That is the thesis of this post: the failures that actually kill teleoperation datasets are invisible in any single episode and only show up in aggregate, so the fix is instrumentation at ingest, not heroics at training time. Each problem looks like a minor engineering detail until it compounds. A 40-millisecond timestamp offset is invisible in a demo video. Spread across 10,000 episodes, it is the difference between a 70 percent policy and a 45 percent one.

Below are the five failures we see most often across the programs we run and audit, each with the fix we actually use. If you are new to the space, start with the complete teleoperation data collection guide and come back.

Key Takeaways – Timestamp skew between 50 Hz actions and 30 fps cameras is the top QA rejection cause we see. Gate at 33 ms, at ingest. – Operator churn is a 3x throughput event, not an HR footnote: trained operators produce 25 to 40 usable episodes per hour versus 8 to 12 for novices. – Reset drift quietly narrows your state distribution; randomization zones fix it. – QA discovered at training time is loss; QA at ingest is feedback. Expect 10 to 30 percent rejection either way, so choose when you find out. – Format decisions are nearly irreversible after 10,000 episodes. Adopt LeRobot conventions on day one.

1. Timestamp Skew: The 50 Hz vs 30 fps Trap

Timestamp skew is the misalignment between a robot’s high-rate action stream and its lower-rate camera streams, and it is the single most common reason episodes fail our QA gates. Actions log at 50 Hz; cameras deliver 30 fps; USB buffering, clock drift between machines, and software timestamps assigned at receipt rather than exposure can push streams 30 to 100 ms apart. A policy trained on skewed data learns that consequences precede actions. It will move confidently and miss.

The fix. Hardware timestamps at the sensor wherever the camera supports it; a single shared monotonic clock otherwise; and an ingest gate that rejects any episode with estimated inter-stream skew above one camera frame (33 ms). We also log a clap-style sync event at the start of each shift as a cheap ground-truth check. Action chunking in ACT-style training tolerates per-step jitter, but no algorithm recovers from systematic offset (arxiv.org/abs/2304.13705).

2. Operator Churn: The Throughput Cliff

Operator churn is the loss of trained teleoperators before their throughput has paid back their ramp cost. The ramp curve is steep: 8 to 12 usable episodes per hour in the first two weeks, 25 to 40 after certification. Every departure resets a station to novice yield for two to three weeks, which means a floor with 20 percent monthly churn is quietly running at two-thirds of its rated capacity.

The fix. Treat operators as skilled technicians, because they are. Certification gates with pay progression, per-operator quality dashboards (people improve what they can see), 45-minute session blocks on VR rigs to manage fatigue, and rig ergonomics worth caring about. Leader-follower stations retain operators better than SpaceMouse benches for the simple reason that the work feels like skill rather than struggle.

3. Reset Drift: The Shrinking State Distribution

Reset drift is the gradual narrowing of initial-state diversity as operators unconsciously converge on the fastest reset, placing objects in the same pose, episode after episode. Nobody notices because every individual episode passes QA. The policy trains beautifully, evaluates beautifully on the collection table, then fails the moment an object appears five centimeters left of its usual spot. This is how teams end up with 10,000 episodes of one initial state.

The fix. Marked randomization zones on the workspace, task cards that specify pose ranges, and a distribution monitor in the QA pipeline that plots initial object positions per task per week. When the scatter tightens, the floor lead hears about it the next morning. Datasets like DROID earned their generalization value from scene diversity, 564 scenes across 76,000 episodes, not from episode count alone (arxiv.org/abs/2403.12945).

4. Silent QA Loss: Finding Out at Training Time

Silent QA loss is the gap between episodes collected and episodes actually usable, discovered only when someone trains on the data weeks later. Rejection is not the problem; 10 to 30 percent rejection is normal and we publish those numbers. The problem is timing. Rejection found at ingest becomes operator feedback and a corrected process by tomorrow. Rejection found at training time is a hole in the dataset, a slipped deadline, and no way to know which of the last six weeks went wrong.

The fix. Nightly automated gates on every episode: sync skew, dropped frames, joint-trajectory jerk thresholds, gripper event sanity, and success-label verification against a sampled human review. Route failures to the specific operator and rig. Our floors run this as a standing morning report, and it is the highest-ROI engineering month a data program can spend.

5. Format Lock-In: The Decision You Make Once

Format lock-in is the practical irreversibility of episode schema decisions once collection is underway, because re-processing tens of thousands of episodes costs more than most teams will ever approve. Missing per-episode metadata, camera intrinsics never recorded, action spaces defined ambiguously (commanded versus measured joint positions, unlabeled): each omission is trivial to fix on day one and a migration project at episode 20,000.

The fix. Adopt community conventions before the first production episode: LeRobot dataset format for training and sharing (github.com/huggingface/lerobot), HDF5 or ROS 2 bags at capture (docs.ros.org), both commanded and measured joint states recorded, camera intrinsics and extrinsics stored per rig per calibration, and schema versioning from v0. Fifteen minutes of schema review now, or fifteen days of migration later.

How to Audit Your Own Pipeline This Week

A pipeline audit is a half-day exercise that estimates your exposure to all five hidden challenges using data you already have. No new tooling required, in most cases.

Pull ten random episodes from last month and compute the maximum camera-to-control timestamp offset per episode; anything over 33 ms means challenge one is live in your dataset right now. Ask your floor lead for usable-episodes-per-hour by operator tenure; if you cannot get the number, that absence is itself the finding. Scatter-plot initial object positions for your highest-volume task across weeks one and four of collection; visibly tightening scatter is reset drift in progress. Check when rejection is discovered: if the answer involves a training run, you have silent QA loss by definition. And finally, hand an episode file to an engineer who has never seen your schema and time how long it takes them to load actions, states, and calibrated camera streams; over thirty minutes predicts format pain at scale.

Every one of these checks has caught a live problem on programs we have audited, including our own. The five challenges are not exotic failure modes; they are the default state of a pipeline that nobody has audited yet.

The Checklist

Hidden challenge Detection signal Fix Cost of ignoring
Timestamp skew Skew estimate > 33 ms at ingest Hardware timestamps + sync gate Policies that act early/late; top rejection cause
Operator churn Throughput below 20 eps/hr floor-wide Certification, pay progression, ergonomics Floor runs at ~2/3 capacity
Reset drift Initial-state scatter tightening week over week Randomization zones + distribution monitor Policy memorizes one layout
Silent QA loss Rejection discovered at training time Nightly ingest gates + operator feedback 10 to 30% of budget wasted invisibly
Format lock-in Schema questions at training time LeRobot conventions + schema v0 Migration project at episode 20k

Next Step

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the most common QA failure in teleoperation data?

Inter-stream timestamp skew, typically between 50 Hz action logs and 30 fps camera streams. We gate at 33 ms of estimated skew and it remains the top rejection reason on new rigs.

Expect 20 to 30 percent rejection during the first two weeks on a new task or rig, settling to 10 to 15 percent in a mature pipeline. Budget the buffer into episode targets.

Usually reset drift: operators converge on repeated initial states during collection, so the dataset covers a narrow state distribution. Randomization zones and initial-state monitoring prevent it.

At ingest, nightly, never later. Rejection found at ingest becomes next-day operator feedback; rejection found at training time is unrecoverable loss weeks after the fact.

LeRobot dataset conventions for training and sharing, with HDF5 or ROS 2 bags at capture. Record both commanded and measured joint states, camera calibration per rig, and version the schema from the start.

5 Hidden Costs in Robot Training Data Budgets (and How to Solve Each One)

Posted on by Sainath Gupta

Two teams we worked alongside last year signed for near-identical manipulation datasets at headline rates within a few dollars of each other. One closed its program roughly on budget. The other overran by nearly half and cut a planned training run to pay for it. The difference was never the rate. It was five quieter line items that neither quote itemized, and only one team went looking for them before signing.

I review DexSet’s QA ledgers, so I watch this pattern from the inside, and it supports one claim, which is the thesis of this post: data budgets die in the gap between quoted raw hours and delivered usable hours, and the five costs below are that gap, itemized. Teams negotiate hard on the dollars-per-hour figure, sign, and then watch these quieter items add 30 to 80 percent to the program. By the time the overrun is visible, the training run is scheduled and there is no negotiating position left.

These five costs stay hidden for a structural reason: they live in the gaps between quote and delivery. A quote prices raw hours; delivery is measured in usable, annotated, retrievable hours. Everything between those two definitions is where money leaks, and vendors have little incentive to itemize a gap that flatters their pricing.

This post names the five, puts our real numbers on each, and gives you the fix. All figures come from DexSet’s own pipeline benchmarks and program ledgers; where public hardware anchors exist, like the roughly $20k ALOHA rig, they are cited.

Read it before you sign anything. Then take the checklist at the bottom into your next vendor call.

Key Takeaways

  • QA rejection (10 to 30% in our pipelines) is the largest hidden cost: a $40 quote at 25% rejection is really $53.33 per usable hour.
  • Annotation scope creep adds $8 to $25 per hour per pass; label only selected data, not the whole corpus.
  • Rig downtime and recalibration silently cut utilization; every idle shift raises amortization per hour.
  • Storage, egress, and versioning run 3 to 6% of capture spend and spike at training time.
  • Operator turnover resets a 30 to 50% throughput learning curve; retention is a data-cost lever.

1. QA Rejection: The Gap Between Raw Hours and Usable Hours

QA rejection cost is the money spent capturing episodes that never enter your training set, and it is invisible in any quote expressed per raw hour. In our pipelines, 10 to 30 percent of raw episodes fail review: dropped frames, desynchronized views, occluded end-effectors, failed task completions.

The math bites harder than teams expect:

Quoted rate Rejection rate Real cost per usable hour Hidden premium
$30 10% $33.33 +11%
$40 25% $53.33 +33%
$50 30% $71.43 +43%

The fix. Three contract clauses. One: the vendor reports a measured rejection rate from a comparable program, not an aspiration. Two: recollection of rejected episodes is priced in writing, ideally on the vendor’s account above an agreed threshold. Three: you run a 100-to-200-hour paid pilot scored against your acceptance spec before any volume commitment. On our programs, a versioned task spec alone typically pulls rejection from the high 20s to under 15 percent within a few weeks.

2. Annotation Scope Creep: Paying for Labels You Never Train On

Annotation scope creep is the gradual expansion of labeling passes across an entire corpus when only a subset of the data needs them. Each pass costs real money: $8 to $12 per data hour for language instructions, $10 to $16 for subtask segmentation, $18 to $25 for dense masks and contact labels, per our benchmarks.

The failure mode is ordering “full annotation” on day one, before anyone knows which slices the model will actually consume. A 10,000-hour corpus with three blanket passes at a mid-range $38 per hour combined is $380,000 of labels, and in our experience a meaningful fraction of densely labeled episodes never influence a training run.

The fix. Stage it. Label language instructions broadly if your VLA needs them, then gate expensive passes behind data selection: annotate the episodes your curriculum actually samples. Run labeling in tranches with a two-week lag behind training experiments, so label spend follows demonstrated need. Teams that stage annotation typically spend 40 to 60 percent less on labels for the same eval performance.

3. Rig Downtime and Recalibration: Utilization Is the Denominator

Downtime cost is the amortization you pay while a rig is not collecting: maintenance, recalibration, resets, and idle shifts all raise the hardware cost of every hour that does get captured. A roughly $20k ALOHA-class cell (https://arxiv.org/abs/2304.13705), or a $32k Mobile ALOHA (https://arxiv.org/abs/2401.02117), is cheap only when it runs.

The numbers move fast. At two-shift utilization over 18 months, amortization is $4 to $9 per hour. Single shift with 50 percent idle time, and the same rig charges you $15 or more per hour before anyone touches a leader arm. Multi-camera exocentric arrays are worse offenders: calibration after every scene change eats 5 to 15 percent of scheduled collection time if it is not engineered out.

The fix. Treat utilization as a weekly KPI. Schedule calibration and scene resets into shift handovers, keep spare grippers and cameras on the shelf (a $600 spare beats a lost shift), and pre-stage scenes so operators walk into ready cells. If you are buying rather than building, ask the vendor how many shifts their rigs run; their utilization sets the amortization share baked into your rate.

4. Storage, Egress, and Versioning: The Bill That Arrives at Training Time

Data infrastructure cost is the spend on storing, versioning, and moving your dataset, and it stays invisible until the first big training run pulls the whole corpus out of cloud storage. Multi-view stereo capture generates terabytes per week; a 10,000-hour multi-camera program can produce several hundred terabytes before compression decisions are made.

Our planning figure is 3 to 6 percent of capture spend for storage, format conversion, and dataset versioning, with egress as the spike risk: pulling a few hundred terabytes across clouds at list egress prices can add tens of thousands of dollars per full-corpus read.

The fix. Decide storage format and residency before collection starts, not after. Co-locate data with training compute to kill egress. Standardize on a training-ready format on delivery (for example, LeRobot-compatible datasets, https://github.com/huggingface/lerobot, rather than raw ROS bags), so you pay conversion once. Version at the episode level so experiments pull slices, not the whole corpus.

5. Operator Turnover: The Learning Curve You Pay For Twice

Operator turnover cost is the throughput and quality you lose when a trained teleoperator leaves and a new one restarts the learning curve. In our programs, operators improve 30 to 50 percent in episodes-per-shift over their first 200 hours, and their rejection rates fall in parallel. Every departure resets both curves.

This cost hides inside blended rates. A vendor churning operators quietly delivers you a workforce that is permanently early-curve: slower, more rejected episodes, same invoice. You will never see a line item for it.

The fix. Ask vendors for operator tenure and how many hours their median operator has logged. In-house, pay experienced operators above generic labor rates; the throughput math justifies it easily. And instrument per-operator metrics, episodes per shift and rejection rate, so coaching happens before quality drifts.

The Pre-Signature Checklist

A pre-signature checklist converts these five hidden costs into questions a vendor must answer in writing before you commit volume:

  • ☐ Measured QA rejection rate on a comparable program, and who pays for recollection
  • ☐ Itemized rate card: capture, QA, each annotation pass ($8 to $25/hr range), infrastructure
  • ☐ Rig utilization (shifts per day) behind the amortization in the rate
  • ☐ Delivery format, storage residency, and who pays egress
  • ☐ Median operator tenure and hours logged
  • ☐ 100-to-200-hour paid pilot scored against your acceptance spec

If a vendor stalls on more than one of these, the hidden costs are not hidden from them. They are hidden from you. The full rate benchmarks behind every number in this post are published in our robot training data costs and pricing guide.

Audit Your Next Quote Against These Five

Ready to scope your data program? Talk to our team.

Request a Quote

Frequently Asked Questions

What is the single biggest hidden cost in robot training data?

QA rejection. At the 10 to 30 percent rejection rates we measure, a quoted raw-hour rate understates the true cost per usable hour by 11 to 43 percent. It is the first number to demand from any vendor.

$8 to $25 per data hour per pass in our benchmarks: language instructions at the low end, dense masks and contact labels at the top. Staging annotation behind data selection typically cuts label spend 40 to 60 percent.

Plan 3 to 6 percent of capture spend, and engineer egress out by co-locating data with training compute. Multi-view stereo programs can reach hundreds of terabytes, so format and residency decisions belong before collection starts.

Throughput improves 30 to 50 percent over an operator’s first 200 hours and rejection falls in parallel. High-churn workforces deliver permanently early-curve performance at the same hourly rate.

Use the six-question checklist above: measured rejection rate, itemized rates, rig utilization, delivery format and egress liability, operator tenure, and a paid pilot against your spec.

5 Hidden Challenges in Exocentric & Multi-View Data for Robot Learning (and How to Solve Them)

Posted on by Sainath Gupta

Multi-view data is usually defined as episodes recorded from several calibrated, synchronized cameras. That definition hides the real problem: calibration and synchronization are not properties you establish, they are properties you maintain, and every mechanism that breaks them is invisible on the day it happens. A dead camera stops a capture session; everyone notices, everyone fixes it. A tripod nudged 4 millimeters by a cleaning crew stops nothing. Capture continues for three weeks, the extrinsics in the metadata quietly stop matching reality, and the first symptom appears a month later as a training curve that will not converge, in a building the data team does not work in.

That gap between when multi-view data breaks and when anyone finds out is what makes these challenges “hidden,” and it is the thesis of this post: detection latency, not defect rate, is what actually determines the cost of a multi-view failure. Single-view pipelines fail loudly. Multi-view pipelines add cross-camera dependencies (shared clocks, relative poses, matched exposure) that fail silently, and the cost of the failure scales with how long it goes undetected.

I run data quality at DexSet, which means my job is finding these failures before customers do. This post covers the five that account for nearly every multi-view escalation we have handled, each with the fix we actually use, not the fix that sounds good in a slide. Rig geometry, camera selection, and cost context live in the pillar: The Complete Guide to Exocentric & Multi-View Data for Robot Learning.

Key Takeaways – The five silent killers: calibration drift, time sync skew, storage blowup, viewpoint overfitting, and lost extrinsics metadata. – Every fix is procedural, not exotic: per-session verification sweeps, PTP or hardware triggering, codec decisions before capture, view dropout in training, and extrinsics embedded per episode. – Our gates: reprojection error under 0.5 px and cross-camera skew under 10 ms, checked every session. Sessions that fail do not ship. – Detection latency is the real enemy. A defect caught same-day costs minutes; caught post-training, it costs a recollection.

Challenge 1: Calibration Drift

Calibration drift is the gradual or sudden divergence between a rig’s recorded extrinsics and its physical camera poses, caused by bumped mounts, sagging booms, thermal cycles, or fixture swaps. It is the most common defect we see in incoming third-party datasets, and the most expensive, because every frame captured after the drift event carries wrong geometry.

The failed solution is calibrating on install and trusting the mounts. Mounts do not deserve trust. Our fix has three parts: rigid fixturing over convenient fixturing (no friction-arm boom mounts on anything that ships), a 20-second ChArUco verification sweep at the start of every session, and a hard ingestion gate at 0.5 px reprojection error. The sweep costs two minutes. In one recent engagement it caught a sagged mount at 1.3 px the morning after a fixture swap; 25 minutes of recalibration instead of weeks of poisoned data. The toolchain is standard: OpenCV calibration for intrinsics and stereo pairs, Kalibr for full multi-camera rigs, ROS 2 camera_calibration for live-topic workflows.

Challenge 2: Time Synchronization Skew

Sync skew is disagreement between cameras about when a frame was captured, and it turns multi-view supervision into contradiction: two views of “the same instant” showing the gripper in two different places. At 30 fps, a one-frame offset is 33 ms of world-state disagreement, injected into every training sample, unremovable after the fact.

Software timestamps drift because independent device clocks drift; that is not a bug, it is physics. The fix is choosing a sync tier deliberately. PTP (IEEE 1588) disciplines clocks over Ethernet to sub-millisecond agreement and is the right default for GigE cameras. Hardware trigger lines or genlock drive shutters from one signal and are mandatory for fast motion. Consumer RGB-D units vary: the RealSense D455 exposes a sync pin, the OAK-D accepts GPIO triggering, the ZED 2i relies on timestamps. Two operational rules: put every sensor on the same clock domain, including wrist cameras running off arm controllers (we once shipped a pilot with an 18 ms wrist offset from exactly that mistake), and verify skew every session against a shared event, gated at 10 ms for manipulation work.

Challenge 3: Storage and Throughput Blowup

The storage blowup is the 3x to 4x multiplication of data volume that multi-view imposes on every downstream system: disks, network links, ingestion jobs, QA review time, and cloud egress bills. A 4-camera rig at 1080p30 produces 0.8 to 1.5 TB per capture day in our pipelines. Teams that budgeted storage for single-view discover this at the worst possible moment, mid-program.

The fix is making format decisions before the first capture day. Choose the codec deliberately (we deliver H.265 for RGB streams; raw or lightly compressed depth where the training plan needs it), define retention tiers so raw footage does not live on hot storage forever, and, most importantly, decide camera count with an ablation instead of ambition. DROID shipped roughly 76,000 episodes with three views, not eight (arXiv:2403.12945); in our own ablations the fourth camera adds about 25 percent to storage and QA cost while rarely moving policy success.

Challenge 4: Viewpoint Overfitting

Viewpoint overfitting is a policy’s silent dependence on the exact camera poses it was trained with, so that a deployment camera mounted 10 cm off the training pose degrades performance without any visible domain shift. It is the mirror image of the problem multi-view was supposed to solve, and fixed rigs make it worse: perfectly consistent extrinsics are perfectly consistent spurious features.

The solution operates on both sides of the pipeline. On capture: introduce controlled pose variation across stations and sessions (a few degrees and centimeters, logged in the extrinsics, never random and unrecorded). On training: view dropout and view shuffling, so the model cannot rely on any single camera being present, a practice consistent with RoboMimic’s finding that observation space composition materially shifts outcomes (arXiv:2108.03298). Ego-Exo4D’s paired design (arXiv:2311.18259) shows the research-grade version: cross-view correspondence learned explicitly rather than avoided.

Challenge 5: Lost Extrinsics and Metadata Decay

Metadata decay is the separation of calibration and sync records from the episodes they describe, until the dataset becomes multi-view footage with no usable geometry. The classic form: extrinsics live in a wiki page or a calib_final_v3.yaml on someone’s laptop, the rig changes, the file does not, and two years later nobody can say which episodes match which calibration. Much of the viewpoint friction in aggregated corpora like Open X-Embodiment (arXiv:2310.08864) traces back to exactly this: heterogeneous sources where camera geometry was never packaged with the data.

The fix is structural: embed per-camera intrinsics, extrinsics, sync offsets, and the calibration verification result inside every episode’s metadata, written by the capture software, not by a human. An episode should be self-describing forever. We also version calibrations with timestamps and station IDs, so any historical episode can be traced to the exact calibration event that produced its geometry.

The Five Challenges at a Glance

Challenge Silent Symptom Fix Our Gate
Calibration drift Training noise weeks later Per-session ChArUco sweep, rigid mounts < 0.5 px reprojection error
Sync skew Cross-view label contradiction PTP / hardware trigger, shared clock domain < 10 ms cross-camera skew
Storage blowup Budget collapse mid-program Codec + retention decided pre-capture; ablate camera count ~1 TB/day per 4-cam rig planned
Viewpoint overfitting Deployment-time degradation Logged pose variation + view dropout in training Pose deltas recorded per session
Metadata decay Unusable geometry years later Extrinsics embedded per episode, versioned Episode self-describing, machine-written

Audit Your Rig Before It Audits You

Every one of these five failures has a one-line question that exposes it in a vendor conversation, and we have collected them into the Multi-View Rig RFP Scorecard. Download it before your next data procurement call, or use it to audit your own rig. Either way, ask the reprojection error question first.

Request a Quote

Frequently Asked Questions

What is the most common failure in multi-view robot datasets?

Calibration drift: physical camera poses diverging from recorded extrinsics after bumps, sag, or fixture changes. It is silent at capture time and surfaces later as training noise, which is why per-session verification with a reprojection error gate is the standard fix.

For tabletop manipulation at 30 fps, DexSet gates at 10 ms cross-camera skew; a full frame of offset (33 ms) injects visible world-state contradiction into training pairs. Fast or dynamic tasks need hardware triggering rather than timestamp alignment.

Combine controlled, logged pose variation across capture sessions with view dropout during training, so the model cannot depend on any exact camera pose or on every view being present.

Plan for 3x to 4x your single-view volume: roughly 0.8 to 1.5 TB per capture day for a 4-camera 1080p30 rig with H.265 RGB, more if depth streams are kept raw. Decide codecs and retention tiers before capture begins.

Inside every episode’s metadata, written automatically by the capture software: per-camera intrinsics, extrinsics, sync offsets, and the session’s verification result. External calibration files decay; self-describing episodes do not.

5 Hidden Challenges in Egocentric Data Collection for Robotics and How to Solve Them

Posted on by Enoch Pakanati

Aviation stopped losing aircraft to dramatic causes decades ago; what remained were the quiet ones, a skipped checklist item, an ambiguous callout, a gauge that read normal until the moment it mattered, and the industry’s answer was to treat small procedural failures as the primary safety problem rather than a footnote. Egocentric data collection is living through the pre-checklist era of that same history. Budgets and rig choice get argued about in every kickoff meeting, so they get managed. The hours that actually get rejected in my QA queue fail for quieter reasons: a timestamp that drifted 80 milliseconds, an auto-exposure loop hunting under warehouse LEDs, a bystander’s face where a consent form should be.

These failures stay hidden because each one is invisible at capture time. The collector sees a normal recording. The file uploads fine. The problem surfaces weeks later, when an engineer tries to align IMU packets with video frames, or a lawyer asks who signed off on the person in frame 40,000. By then the collection trip is over and the fix is a re-shoot. That is this post’s thesis: the failures that destroy egocentric datasets are procedural and invisible at capture time, so only enforced process, not better hardware, catches them before they compound.

I review egocentric footage for a living, roughly a few thousand hours a quarter across DexSet’s programs. This post covers the five failure modes that cost our clients (and, in the early days, us) the most money, what each one does to your training data, and the fix we now enforce by default. Every one of these is checkable before you scale, and none of them appears on a hardware spec sheet.

Key Takeaways – The expensive failures in egocentric collection are invisible at capture time: sync drift, exposure instability, bystander privacy, protocol drift, and metadata chaos. – Sensor sync should be verified per session, not trusted; drift beyond ~10 ms between IMU and frames degrades ego-motion estimation and any action retargeting built on it. – Auto-exposure and rolling shutter quietly poison fast-motion segments; lock exposure ranges per environment and test at task speed. – Privacy is a data-quality gate, not just a legal one; footage you cannot license cleanly is footage you cannot ship, whatever its pixel quality. – Fixing these adds roughly $2 to $5 per hour in process cost and saves multiples of that in rejected hours; our first-pass yields moved from ~70 to 85+ percent after enforcing the five fixes below.

Challenge 1: Sensor Sync Drift

Sensor sync drift is the accumulating misalignment between timestamps of different streams (video frames, IMU packets, depth, gaze) recorded on the same rig. Consumer devices are the usual culprits: separate clocks, buffered writes, and dropped packets each add error, and error compounds over a 40-minute session.

Why it hurts: ego-motion estimation fuses IMU with frames, and hand-trajectory extraction inherits any misalignment as velocity noise. Once drift passes roughly 10 ms, we see it in reprojection error; past 30 ms, retargeting-grade annotation is off the table for that session. Purpose-built devices earn their price here; Aria-class glasses timestamp all streams against a common clock with factory calibration (projectaria.com), which is a large part of why their finished-hour costs are competitive despite the hardware premium.

The fix: a physical sync event at every session start and end (a clap or strobe visible to all sensors), automated drift measurement between the two events, and a hard QA gate. On custom RealSense rigs, we also pin capture to hardware timestamps rather than host arrival times. Cost: seconds per session and one script.

Challenge 2: Auto-Exposure and Motion Blur at Task Speed

Auto-exposure instability is the camera’s exposure control oscillating or lagging as the wearer’s head sweeps between bright and dark regions, and it is the single largest source of rejected kitchen and warehouse footage in our logs. Head-mounted cameras experience scene luminance changes far faster than handheld ones. Add rolling shutter and fast hand motion and you get smeared, flickering segments exactly where the manipulation happens.

The insidious part is that slow test recordings look fine. Collectors validate the rig while standing still; the failure only appears at task speed, and encoders trained on smeared hands learn smeared hands.

The fix: per-environment exposure presets (shutter-priority for fast tasks), a five-minute test capture at full task speed before every new location, and an automated blur metric in the QA pipeline. On GoPro-class rigs we run higher frame rates (60 to 120 fps) to buy shutter headroom. This one protocol change moved a client program’s first-pass yield by eight points.

Challenge 3: Bystander Privacy and Consent

Bystander capture is the recording of people who never consented to be in your dataset, and in egocentric collection it is the default outcome rather than the exception, because the camera goes wherever the collector’s head turns. Large academic efforts treated this as a first-class problem; Ego4D built consent processes and de-identification into its collection across 74 locations (arXiv:2110.07058), and commercial programs are held to a stricter standard, not a looser one.

Treat this as data quality, not legal overhead: an hour you cannot license is an hour you cannot deliver, and face-blurring after the fact degrades exactly the frames where a person interacts with the scene.

The fix: controlled capture sites with posted notice and signed consent for everyone who could enter frame, collector training to pause on unexpected entries, and automated face/plate detection as a QA flag rather than a silent blur. Where blurring is unavoidable, we log it in episode metadata so downstream users can filter.

Challenge 4: Protocol Drift Across Collectors and Weeks

Protocol drift is the slow divergence between what your task specification says and what collectors actually do by week six. Human demonstrators optimize for their own comfort: grasps migrate to five-finger tricks a parallel-jaw gripper cannot copy, “reset the scene” becomes approximate, and each site develops a local dialect of the task. None of this looks like an error in any single episode. In aggregate, it shifts your action distribution away from anything a robot can imitate, widening the embodiment gap that retargeting work like EgoMimic goes to such lengths to close (arXiv:2410.24221).

The fix: written task cards with start-state photos and outcome rules, motion constraints stated in robot terms (gripper envelope, reach profile), weekly sampled review per collector rather than per batch, and re-training triggered by drift metrics instead of complaints. Episode-structured schemas, like the conventions the LeRobot ecosystem standardized (github.com/huggingface/lerobot), make drift measurable because every episode declares what it claims to contain.

Challenge 5: Metadata Chaos

Metadata chaos is the absence of a consistent schema binding each recording to its calibration, rig identity, task card, collector, consent record, and QA status, and it is the challenge teams discover last because nothing fails until you scale. At 100 hours, a spreadsheet works. At 4,000 hours across 26 collectors and two rig families, a spreadsheet is how you lose track of which calibration file belongs to which helmet after a lens swap.

The downstream symptom is quiet and awful: models trained on hours whose intrinsics are wrong by one rig revision. Cross-institution efforts like Open X-Embodiment succeeded precisely because they forced heterogeneous data into a common, self-describing format (arXiv:2310.08864).

The fix: schema before scale. Every episode carries rig serial, calibration hash, protocol version, consent reference, and QA verdict as first-class fields, validated at ingestion, so a bad calibration recall touches exactly the affected hours and no others.

The Checklist

Hidden Challenge Detection Fix Process Cost
Sync drift Clap/strobe events + automated drift check Common-clock rigs or hardware timestamps; hard QA gate at 10 ms ~$0.50/hr
Exposure/blur Blur metric on every batch Per-site presets, task-speed test capture, 60 to 120 fps ~$1/hr
Bystander privacy Face/plate detection flags Controlled sites, consent workflow, logged blurs $1 to $2/hr
Protocol drift Weekly per-collector sampling Task cards, robot-term constraints, drift-triggered retraining $1 to $2/hr
Metadata chaos Ingestion-time schema validation Episode-level schema with calibration hash and consent reference ~$0.50/hr

Total: roughly $2 to $5 per hour of process cost. Against a 15 to 25 percent rejection rate on unmanaged programs, it is the cheapest insurance in robotics data. The full economics, rig comparisons, and program design context live in our pillar guide: The Complete Guide to Egocentric Data Collection for Robotics.

Take the Checklist With You

We turned this post into a one-page QA checklist our own collectors carry. Download it, tape it to your rig case, and if you would rather someone else run the checklist for you, that is literally our job.

Request a Quote

Frequently Asked Questions

What is the most common quality failure in egocentric robotics data?

In DexSet’s QA logs, exposure and motion-blur failures at task speed are the most common rejection cause, followed by sensor sync drift between IMU and video streams.

We gate at roughly 10 milliseconds for retargeting-grade data; beyond 30 milliseconds, hand-trajectory extraction becomes unreliable for that session.

Controlled capture sites with posted notice and signed consent, collector training to pause on unexpected entries, automated face detection as a QA flag, and metadata records for any blurred segments.

Protocol drift is collectors gradually diverging from the task specification, such as using grasps a robot gripper cannot reproduce. It shifts the dataset’s action distribution away from the robot embodiment and degrades retargeting and co-training value.

Roughly $2 to $5 per hour in process cost across sync checks, blur metrics, privacy workflow, drift sampling, and schema validation, which typically pays for itself by lifting first-pass yield from around 70 percent to 85 percent or higher.