fix(dataset): use revision-safe Hub cache for downloaded datasets

[AdilZouitine]

What & Why

LeRobotDataset currently calls snapshot_download(local_dir=self.root) for every
Hub fetch, which flattens all revisions into a single directory tree under
$HF_LEROBOT_HOME/{repo_id}. When two processes (or two training jobs) read
different revisions of the same dataset concurrently, one overwrites the other's
files, corrupting both.

Fixes #3224

How

When root=None (the default for all Hub-downloaded datasets), downloads now use
the Hub's native content-addressable blob store:

snapshot_download(repo_id, cache_dir=HF_LEROBOT_HUB_CACHE, revision=...)

instead of the previous:

snapshot_download(repo_id, local_dir=self.root, revision=...)

Each revision resolves to its own immutable snapshot directory under
$HF_LEROBOT_HOME/hub/datasets--{org}--{name}/snapshots/{commit}/, so two
concurrent readers can never step on each other. Blobs are shared via hardlinks,
so disk usage stays flat (~93 MB for a 30-episode dataset regardless of how many
revisions are checked out).

When root is explicitly provided (local datasets, recording, etc.), the
existing local_dir materialization is preserved — no behavioral change.

A lightweight heuristic (_has_legacy_hub_download_metadata) detects old
local_dir mirrors (they contain .cache/huggingface/download/) and
transparently migrates to the snapshot cache on the next load.

Files changed:

File	Summary
configs/default.py	Updated docstring for root field
datasets/dataset_metadata.py	Route _pull_from_repo through cache_dir when root=None; add legacy-mirror detection
datasets/lerobot_dataset.py	Route _download through cache_dir when root=None; propagate resolved root to reader/meta
datasets/streaming_dataset.py	Mirror the same cache_dir routing for the streaming path
utils/constants.py	Add HF_LEROBOT_HUB_CACHE constant
tests/datasets/test_lerobot_dataset.py	3 new tests covering snapshot isolation, cache-dir routing, and legacy bypass

Testing

Manual concurrent-read test (reproducer from the issue):

# Terminal 1 — reads "main" (has 'test' column)
python experiment_concurent_read.py main

# Terminal 2 — reads an older commit (no 'test' column)
python experiment_concurent_read.py b59010db93eb6cc3cf06ef2f7cae1bbe62b726d9

Before this fix both terminals eventually report *** CHANGED *** as one process
overwrites the other's parquet files. After this fix each terminal reads from its
own snapshot directory and the columns never change:

# Terminal 1 (main);  always True
[09:39:13] [main] has 'test' column=True

# Terminal 2 (old commit); always False
[09:39:26] [b59010db93eb] has 'test' column=False

Cache stays compact ; the Hub blob store deduplicates unchanged files via hardlinks:

❯ du -sh datasets--AdilZtn--pick_and_place_intermediate_30_ep
 93M    datasets--AdilZtn--pick_and_place_intermediate_30_ep
❯ du -sh datasets--AdilZtn--pick_and_place_intermediate_30_ep/snapshots/
  0B    datasets--AdilZtn--pick_and_place_intermediate_30_ep/snapshots/

Unit tests:

pytest -sx tests/datasets/test_lerobot_dataset.py::test_metadata_without_root_uses_hub_cache_snapshot_download
pytest -sx tests/datasets/test_lerobot_dataset.py::test_without_root_reads_different_revisions_from_distinct_snapshot_roots
pytest -sx tests/datasets/test_lerobot_dataset.py::test_metadata_without_root_ignores_legacy_local_dir_cache

Notes for reviewers

• The core change is small: _pull_from_repo and _download branch on
  self._requested_root is None to choose between cache_dir and local_dir.
  Everything else is docstrings, constants, and tests.
• The legacy-mirror heuristic checks for
  root / ".cache" / "huggingface" / "download" — this directory is created by
  snapshot_download(local_dir=...) but never by user-created datasets.
• streaming_dataset.py follows the same pattern for consistency.
• The _requested_root attribute distinguishes "user explicitly set root" from
  "we derived root from the default".

pytest -sx tests/datasets/test_lerobot_dataset.py -k "hub_cache or legacy or distinct_snapshot"

[imstevenpmwork]

Great issue description and follow-up PR, thanks!
I superseded in here for some small details to avoid the back-and-forth: #3233
However; we would need to wait for #3231 to get the CI check green