feat(infra): add blob storage lifecycle policies and folder structure

[rezatnoMsirhC]

feat(infra): add blob storage lifecycle policies and folder structure

Description

Implements Azure Blob Storage folder structure with automated lifecycle management for raw ROS bags, converted LeRobot datasets, validation reports, and model checkpoints. The PR adds a Terraform azurerm_storage_management_policy resource with three configurable rules, a Python path validator that enforces naming conventions across all four data types, and comprehensive documentation describing the folder structure, lifecycle policies, and migration guidance for existing paths.

The lifecycle policy was applied to the IAI-DEV environment and all three rules were confirmed via Azure CLI. The Python validator was exercised against test blob paths uploaded to the deployed container, with all 8 naming-convention checks passing alongside the 34-test unit suite.

Closes #45

Type of Change

• 🐛 Bug fix (non-breaking change fixing an issue)
• ✨ New feature (non-breaking change adding functionality)
• 💥 Breaking change (fix or feature causing existing functionality to change)
• 📚 Documentation update
• 🏗️ Infrastructure change (Terraform/IaC)
• ♻️ Refactoring (no functional changes)

Component(s) Affected

• deploy/000-prerequisites - Azure subscription setup
• deploy/001-iac - Terraform infrastructure
• deploy/002-setup - OSMO control plane / Helm
• deploy/004-workflow - Training workflows
• src/training - Python training scripts
• docs/ - Documentation

Testing Performed

• Terraform plan reviewed (no unexpected changes)
• Terraform apply tested in dev environment
• Training scripts tested locally with Isaac Sim
• OSMO workflow submitted successfully
• Smoke tests passed (smoke_test_azure.py)

Additional testing steps:

Unit tests:

.venv/bin/python -m pytest tests/common/test_blob_path_validator.py -v
# 34/34 passed

Formatting and linting:

terraform fmt -check -recursive deploy/
npm run lint:md       # 0 errors (91 files)
npm run spell-check   # 0 issues (401 files)

Terraform plan and apply (IAI-DEV environment):

source deploy/000-prerequisites/az-sub-init.sh
cd deploy/001-iac
terraform init
terraform plan -var-file=terraform.tfvars \
  -target='module.platform.azurerm_storage_account.main' \
  -target='module.platform.azurerm_storage_container.main["ml-workspace"]' \
  -target='module.platform.azurerm_storage_management_policy.main'
# Plan: 4 to add, 0 to change, 0 to destroy
terraform apply -var-file=terraform.tfvars \
  -target='module.platform.azurerm_storage_account.main' \
  -target='module.platform.azurerm_storage_container.main["ml-workspace"]' \
  -target='module.platform.azurerm_storage_management_policy.main' \
  -auto-approve

Lifecycle policy verification:

az storage account management-policy show \
  --account-name stphysaidev001 \
  --resource-group rg-cmontazer-phys-ai-toolchain-issue-45
# Confirmed: 3 rules — delete-raw-bags (30d), tier-converted-datasets-to-cool (90d),
#            tier-reports-to-cool-then-archive (30d cool / 180d archive)

Blob upload and naming convention verification:

# Assign Storage Blob Data Contributor (required for data-plane access)
az role assignment create \
  --assignee 852213b9-a4bf-4cb6-bdc9-d2cf0057c262 \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-cmontazer-phys-ai-toolchain-issue-45/providers/Microsoft.Storage/storageAccounts/stphysaidev001"

# Upload test blobs for each path type
az storage blob upload --account-name stphysaidev001 --container-name ml-workspace \
  --name "raw/test-device-01/2026-03-12/test-episode.mcap" --file test-episode.mcap
az storage blob upload --account-name stphysaidev001 --container-name ml-workspace \
  --name "converted/test-dataset-v1/meta/info.json" --file info.json
az storage blob upload --account-name stphysaidev001 --container-name ml-workspace \
  --name "reports/test-dataset/2026-03-12/validation-report.json" --file validation-report.json
az storage blob upload --account-name stphysaidev001 --container-name ml-workspace \
  --name "checkpoints/test-policy-v1/20260312_120000.pt" --file checkpoint.pt

# Verify uploads
az storage blob list --account-name stphysaidev001 --container-name ml-workspace \
  --query "[].name" --output tsv
# raw/test-device-01/2026-03-12/test-episode.mcap
# converted/test-dataset-v1/meta/info.json
# reports/test-dataset/2026-03-12/validation-report.json
# checkpoints/test-policy-v1/20260312_120000.pt

Naming convention validation (8/8 checks passed):

from src.common.blob_path_validator import validate_blob_path, get_validation_error

valid_paths = [
    "raw/device-01/2026-03-12/episode.mcap",
    "converted/dataset-v1/meta/info.json",
    "reports/dataset/2026-03-12/report.json",
    "checkpoints/policy-v1/20260312_120000.pt",
]
invalid_paths = [
    "inference_outputs/result.json",   # legacy path
    "raw/episode.mcap",               # missing device/date segments
    "Raw/device-01/episode.mcap",     # uppercase prefix
    "converted/dataset v1/data.json", # space in name
]
# All valid paths accepted, all invalid paths rejected — RESULT: ALL CHECKS PASSED

Documentation Impact

• No documentation changes needed
• Documentation updated in this PR
• Documentation issue filed

docs/cloud/blob-storage-structure.md added (296 lines): folder structure specification, naming conventions, lifecycle policy table, access tier comparison, rehydration instructions, and migration guidance for existing inference_outputs/ and LeRobot paths. README.md updated with a blob storage folder quick-reference.

Bug Fix Checklist

Complete this section for bug fix PRs. Skip for other contribution types.

• Linked to issue being fixed
• Regression test included, OR
• Justification for no regression test:

Checklist

• My code follows the project conventions
• Commit messages follow conventional commit format
• I have performed a self-review
• Documentation impact assessed above
• No new linting warnings introduced

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.50%. Comparing base (d90d66d) to head (73cc647).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #179      +/-   ##
==========================================
+ Coverage   38.37%   38.50%   +0.12%     
==========================================
  Files          73       74       +1     
  Lines        8579     8597      +18     
  Branches      497      501       +4     
==========================================
+ Hits         3292     3310      +18     
  Misses       5277     5277              
  Partials       10       10              

Flag	Coverage Δ		*Carryforward flag
pester	79.87% <ø> (ø)
pytest	7.34% <ø> (+0.44%)	⬆️	Carriedforward from bae4249
pytest-dataviewer	61.98% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[WilliamBerryiii]

Hey @rezatnoMsirhC — great work on this PR! The blob storage lifecycle policies and folder structure look solid, and the validation testing you did was thorough.

I went ahead and resolved the merge conflict by restoring the root README.md to its current main state (commit 52f6b14). The blob storage docs live nicely in docs/cloud/blob-storage-structure.md on their own, so we can add a README reference separately if needed.

Merging shortly — thanks for the contribution! 🚀

[katriendg]

With recent changes it may be your current version has some small inconsistencies.

The one point may be worth adding is ensuring the new test is discoverable, pyproject.toml has testpaths = ["tests", "training/tests"], but is missing the new data-pipeline/capture/tests/ folder.

[katriendg]

Looks good, I believe the re-org of the folders is also captured. Thanks!