feat(dataviewer): enhance HDF5 video handling and nested dataset support

[agreaves-ms]

feat(dataviewer): enhance HDF5 video handling and nested dataset support

This PR strengthens the dataviewer's HDF5 capabilities across video generation, blob storage integration, and dataset organization. It introduces on-demand video generation from HDF5 image data, extends Azure Blob Storage support to HDF5 datasets, enables hierarchical dataset IDs up to 5 levels deep, and simplifies the frontend video playback architecture by removing the persistent frame cache in favor of native HTML5 video rendering.

The frame caching system served its purpose during initial development but added complexity that wasn't justified as HDF5 video generation matured. Native <video> element playback is simpler and more reliable for the generated MP4 files.

Description

HDF5 Video Generation

The HDF5 format handler gained full video generation capabilities with an on-demand caching strategy. Videos are generated using ffmpeg (H.264, ultrafast preset, yuv420p pixel format) with an OpenCV fallback when ffmpeg is unavailable. Generated videos are cached at meta/videos/{camera}/episode_{NNNNNN}.mp4 and uploaded to blob storage when configured.

• Added _generate_video() and _generate_video_cv2() to hdf5_handler.py with graceful degradation between backends
• Implemented get_video_path() with cache-first lookup and synchronous on-demand generation
• Added load_single_frame() in hdf5_loader.py using h5py slice indexing for memory-efficient frame extraction
• Installed ffmpeg system dependency in the backend Dockerfile

Blob Storage HDF5 Integration

Extended the blob dataset provider to discover, sync, and serve HDF5 datasets alongside existing LeRobot support.

• Added scan_all_dataset_ids() for single-pass discovery returning both LeRobot and HDF5 dataset types
• Implemented sync_hdf5_dataset_to_local() for metadata and placeholder sync, and sync_hdf5_episode_to_local() for on-demand episode downloads
• Enhanced video path resolution with _build_video_path_candidates() supporting template-based, index-based, and fallback scan strategies
• Switched from list_blobs() to list_blob_names() for faster blob enumeration
• Added upload_video() for persisting locally generated videos back to blob storage

Nested Dataset ID Support

Enabled hierarchical dataset organization using -- separators (e.g., project--recordings--session_1) mapping to /-separated paths in storage.

• Added paths.py utility with dataset_id_to_blob_prefix() centralized across storage adapters
• Updated _scan_directory() with recursive traversal and _validate_dataset_id() enforcing max 5 segments
• Refactored dataset grouping from split("--", 1)[0] to "--".join(split("--")[:-1]) for correct multi-level grouping
• Updated DataviewerShellHeader.tsx to display group keys with forward slashes instead of dashes

Label Storage Abstraction

Introduced a LabelStorage protocol in labels.py with LocalLabelStorage and BlobLabelStorage implementations, enabling label persistence in both local filesystem and Azure Blob Storage backends. Both implementations include path traversal protection via realpath() + startswith() validation.

Frontend Playback Simplification

Removed the persistent video frame caching system and simplified the annotation workspace playback architecture.

• Deleted useVideoFrameCache.ts (165 lines) and its test suite (244 lines)
• Removed RVFC-based canvas rendering, displayCanvasRef, and cache state management from useAnnotationWorkspaceVideoSync.ts (~150 lines)
• Replaced canvas-based video rendering with direct HTML5 <video> element in AnnotationWorkspacePlaybackCard.tsx
• Added 200ms debounced loading overlay to prevent flicker on quick video loads

Configuration and Environment

• Updated .gitignore dataset path from src/dataviewer/datasets/ to root-level datasets/
• Added .env.azure.example template for Azure Blob Storage development setup
• Enhanced .env.example with Azure CLI prerequisites and permission documentation
• Added HMI_DATA_PATH environment variable to the backend dev script in package.json
• Added Azure SDK logging suppression and lifespan-based temp directory cleanup in main.py

Type of Change

• 🐛 Bug fix (non-breaking change fixing an issue)
• ✨ New feature (non-breaking change adding functionality)
• 💥 Breaking change (fix or feature causing existing functionality to change)
• 📚 Documentation update
• 🏗️ Infrastructure change (Terraform/IaC)
• ♻️ Refactoring (no functional changes)

Component(s) Affected

• deploy/000-prerequisites - Azure subscription setup
• deploy/001-iac - Terraform infrastructure
• deploy/002-setup - OSMO control plane / Helm
• deploy/004-workflow - Training workflows
• src/training - Python training scripts
• docs/ - Documentation

Testing Performed

• Terraform plan reviewed (no unexpected changes)
• Terraform apply tested in dev environment
• Training scripts tested locally with Isaac Sim
• OSMO workflow submitted successfully
• Smoke tests passed (smoke_test_azure.py)

Documentation Impact

• No documentation changes needed
• Documentation updated in this PR
• Documentation issue filed

Bug Fix Checklist

Complete this section for bug fix PRs. Skip for other contribution types.

• Linked to issue being fixed
• Regression test included, OR
• Justification for no regression test:

Checklist

• My code follows the project conventions
• Commit messages follow conventional commit format
• I have performed a self-review
• Documentation impact assessed above
• No new linting warnings introduced

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.47%. Comparing base (745321e) to head (a0e2e1c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #180   +/-   ##
=======================================
  Coverage   37.47%   37.47%           
=======================================
  Files          43       43           
  Lines        6135     6135           
  Branches      497      497           
=======================================
  Hits         2299     2299           
  Misses       3826     3826           
  Partials       10       10           

Flag	Coverage Δ
pester	84.80% <ø> (ø)
pytest	6.89% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.