The physical-AI data loop:
LeRobot and Hugging Face Buckets

Phase 1: collect and ingest

Collection setups record new episodes all day, each a continuous run of camera frames and state-action telemetry, and the LeRobot format appends per episode. But where do those bytes land?

A LeRobotDataset keeps three things: state and action tensors in Parquet, one MP4 stream per camera, and a small meta/ folder with the schema, stats, and per-episode index. Many episodes are packed into a few large shards, so an episode is a slice of a shard located through offset metadata, not a standalone file.

The practical consequence is that a dataset is a few set of large, mutable files that grow as you record. Push them straight to a versioned dataset repository and you fight Git-LFS history: every append is a commit, and every revision is retained forever. During collection you don’t want that. You want a fast place to dump bytes and overwrite freely.

That is a Storage Bucket: a repository type on the Hub that behaves like an S3 bucket, mutable and overwrite-in-place, inside your Hugging Face workspace with your existing permissions (Hugging Face, 2026). No IAM roles, no CORS, no upload service to maintain.

Collection runs append into one mutable place, published repos stay clean, and you wrote no cloud-storage plumbing. The next three phases are where that backend pays you back.

Phase 2: aggregate and deduplicate

Robot data is extraordinarily redundant. When a robotics arm spends eight hours clearing the same table in front of two 1080×720p cameras, the lighting, the chassis, and most of the background stay identical across thousands of episodes. Naive storage pays for all of it every time, and Git LFS makes it worse: change one frame in a multi-gigabyte video shard and it re-uploads the whole file.

Buckets are backed by Xet, which deduplicates at the byte level using content-defined chunking (Hugging Face, 2025; Hugging Face Xet Team, 2025). Chunk boundaries follow the content, so inserting a few bytes changes only the chunk it lands in instead of shifting every boundary after it. Edit the stream below to see it.

For an append-heavy robot dataset, a daily push uploads roughly the new material plus whatever genuinely changed. Successive π0.5 or Gr00t checkpoints with mostly-frozen weights behave the same way (Hugging Face Xet Team, 2025b). A server-side hf buckets cp can even copy a terabyte-scale dataset near-instantly, since it migrates content hashes rather than bytes.

Phase 3: train at scale

Training a VLA like NVIDIA’s Gr00t or a world-action-model like Fast-WAM means pointing GPUs at your datasets. The expensive failure mode is GPUs sitting idle while hundreds of gigabytes download. Pick a mode below to see where the bottleneck moves.

The shard layout from Phase 1 is what makes Hub-native streaming practical: a batch is a few byte-range reads over large shards, not thousands of tiny fetches. StreamingLeRobotDataset turns that into a drop-in torch iterable that trains directly from the Hub, with nothing on local disk:

from lerobot.datasets import StreamingLeRobotDataset

ds = StreamingLeRobotDataset(repo_id="my-org/pick-place")  #streaming=True
for batch in torch.utils.data.DataLoader(ds, batch_size=64):
    loss = policy(batch).loss
    loss.backward()

Pre-warming then caches data at edge locations near the cloud and region where your jobs run, on AWS and GCP, so the cluster reads locally (Hugging Face, 2026). In Hugging Face’s benchmark a bucket served roughly 1,100 MB/s cold and up to about 1,326 MB/s warm, against a generic object store around 310 to 420 MB/s (h-m-t, 2026).

Phase 4: deploy and iterate

A finished run emits a policy checkpoint: roughly a gigabyte for SmolVLA, several for a π0.5-scale model. You push it onto physical robots to evaluate, then pull the next dataset back to improve the policy. The loop only works if moving data is cheap, because you want to do it often. The real version of this already exists: record demonstrations, push them as a LeRobotDataset, validate in simulation, then deploy the same code to hardware by passing a Hub checkpoint id.

lerobot-rollout --policy.path=my-org/pick-place-vla \
  --strategy.type=sentry \
  --robot.type=rebot_b601_follower \

On a self-managed store, the storage rate is the part you see coming. The surprise is egress, the data transferred out of the cloud to your robots or another region: every checkpoint shipped and every dataset pulled is metered per gigabyte, and for a busy fleet it can exceed the storage bill. Hugging Face Storage includes egress and CDN at no extra cost up to an 8:1 ratio of stored volume (Hugging Face, 2026), so for a fleet that ships daily the line item largely disappears. Price your own footprint:

For a mid-sized fleet of YAM arms with 250 TB of recorded episodes, moving 100 TB of egress per month, the private-bucket total is about $42,000/year against roughly $160,000 on self-managed S3, where egress alone is about $93,500. That is close to a 74% reduction (Hugging Face, 2026; Lepers, 2026), and it reproduces from the public cost calculator’s own pricing.

Closing the loop

The four phases share one backend. You collect into a mutable bucket instead of bloating a git repo, Xet uploads and bills only the bytes that changed, StreamingLeRobotDataset and a pre-warmed CDN feed the GPUs without a download stall, and included egress keeps a fleet of robots shipping fresh VLA or World Models checkpoints without running up a transfer bill. None of it asked you to run cloud infrastructure, so your time goes to policies and hardware.

Buckets and LeRobot share the namespace, the Xet backend, and the tooling, which is enough to make them the storage layer the physical-AI loop wants.

curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login

hf buckets create my-org/robot-fave --private

hf sync ./recordings hf://buckets/my-org/robot-fave/run-021

Steven Palma @imstevenpmwork