diff --git a/.github/ISSUE_TEMPLATE/00-general.md b/.github/ISSUE_TEMPLATE/00-general.md index 8762e7d0..9a36343c 100644 --- a/.github/ISSUE_TEMPLATE/00-general.md +++ b/.github/ISSUE_TEMPLATE/00-general.md @@ -54,14 +54,14 @@ Include any relevant information: -| Component | Version | -|-----------|---------------------------| -| OS | Ubuntu 22.04 / Windows 11 | -| Python | 3.10.x / 3.11.x | -| Terraform | 1.9.x | -| Azure CLI | 2.x | -| Isaac Sim | 4.5 / 5.0 | -| GPU | NVIDIA RTX / A100 | +| Component | Version | +|-----------|------------------------------------| +| OS | Ubuntu 22.04 / Windows 11 | +| Python | 3.10.x / 3.11.x / 3.12.x | +| Terraform | 1.9.x | +| Azure CLI | 2.x | +| Isaac Sim | 4.5 / 5.0 | +| GPU | NVIDIA RTX / A100 | ## Additional Notes diff --git a/.github/ISSUE_TEMPLATE/01-bug-report.yml b/.github/ISSUE_TEMPLATE/01-bug-report.yml index c52e1e50..af6e82a9 100644 --- a/.github/ISSUE_TEMPLATE/01-bug-report.yml +++ b/.github/ISSUE_TEMPLATE/01-bug-report.yml @@ -63,7 +63,7 @@ body: description: Provide relevant environment details placeholder: | - OS: Ubuntu 22.04 / Windows 11 - - Python version: 3.10.x / 3.11.x + - Python version: 3.10.x / 3.11.x / 3.12.x - Terraform version: 1.9.x - Azure CLI version: 2.x - Isaac Sim version: 4.5 / 5.0 diff --git a/.github/ISSUE_TEMPLATE/04-training-issue.yml b/.github/ISSUE_TEMPLATE/04-training-issue.yml index 23ecd2a4..a0c3d91b 100644 --- a/.github/ISSUE_TEMPLATE/04-training-issue.yml +++ b/.github/ISSUE_TEMPLATE/04-training-issue.yml @@ -41,7 +41,7 @@ body: id: python-version attributes: label: Python Version - placeholder: "3.11.x" + placeholder: "3.12.x" validations: required: false diff --git a/.github/agents/dataviewer-developer.agent.md b/.github/agents/dataviewer-developer.agent.md index 513218c0..0a54e6d9 100644 --- a/.github/agents/dataviewer-developer.agent.md +++ b/.github/agents/dataviewer-developer.agent.md @@ -202,7 +202,7 @@ Follow these codebase conventions: - New endpoints go in `routers/` (REST) or `routes/` (specialized) - Models in `models/`, services in `services/` - Register new routers in `main.py` -- Use ruff for linting (line-length 120, target py311) +- Use ruff for linting (line-length 120, target py312) **Frontend (React/TypeScript):** diff --git a/.github/skills/dataviewer/SKILL.md b/.github/skills/dataviewer/SKILL.md index b0732c2f..48bbfcab 100644 --- a/.github/skills/dataviewer/SKILL.md +++ b/.github/skills/dataviewer/SKILL.md @@ -11,7 +11,7 @@ Launch and interact with the Dataset Analysis Tool — a full-stack application | Platform | Requirement | |----------|-------------| -| All | Python 3.11+, Node.js 18+, npm, `uv` | +| All | Python 3.12+, Node.js 18+, npm, `uv` | The backend virtual environment and frontend `node_modules` are auto-created on first launch by `start.sh`. @@ -373,7 +373,7 @@ The React app has these key areas for Playwright interaction: | Issue | Solution | |-------|----------| -| Backend fails to start | Check `backend/.venv` exists; run `cd backend && uv venv --python 3.11 && source .venv/bin/activate && uv pip install -e ".[dev,analysis,export]"` | +| Backend fails to start | Check `backend/.venv` exists; run `cd backend && uv venv --python 3.12 && source .venv/bin/activate && uv pip install -e ".[dev,analysis,export]"` | | Frontend shows "Loading..." indefinitely | Verify backend is healthy: `curl http://localhost:8000/health` | | No datasets visible | Check `HMI_DATA_PATH` in `backend/.env` points to a directory with dataset subdirectories | | Port conflict | Set `BACKEND_PORT` or `FRONTEND_PORT` environment variables | diff --git a/.github/skills/osmo-lerobot-training/SKILL.md b/.github/skills/osmo-lerobot-training/SKILL.md index e4bb7a1b..5d5ca9af 100644 --- a/.github/skills/osmo-lerobot-training/SKILL.md +++ b/.github/skills/osmo-lerobot-training/SKILL.md @@ -17,7 +17,7 @@ Read the skill file `.github/skills/osmo-lerobot-training/SKILL.md` for paramete | `az` CLI | Azure authentication and model registry | | `terraform` | Infrastructure output resolution | | `zip`, `base64` | Training payload packaging | -| Python 3.11+ with `azure-ai-ml`, `mlflow` | Metric retrieval from Azure ML | +| Python 3.12+ with `azure-ai-ml`, `mlflow` | Metric retrieval from Azure ML | Authentication must be configured before any OSMO or Azure ML operations: diff --git a/.github/workflows/aw-dependabot-pr-review.md b/.github/workflows/aw-dependabot-pr-review.md index 2f16f919..0df68926 100644 --- a/.github/workflows/aw-dependabot-pr-review.md +++ b/.github/workflows/aw-dependabot-pr-review.md @@ -45,7 +45,7 @@ runtimes: action-repo: actions/setup-node action-version: 53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 python: - version: "3.11" + version: "3.12" action-repo: actions/setup-python action-version: a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 uv: diff --git a/.github/workflows/pytest-data-pipeline.yml b/.github/workflows/pytest-data-pipeline.yml index 7d6607c7..a4bd491f 100644 --- a/.github/workflows/pytest-data-pipeline.yml +++ b/.github/workflows/pytest-data-pipeline.yml @@ -29,7 +29,7 @@ jobs: - name: Setup Python uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: - python-version: '3.11' + python-version: '3.12' - name: Setup uv uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 diff --git a/.github/workflows/pytest-dm-tools.yml b/.github/workflows/pytest-dm-tools.yml index 2c376731..be83890b 100644 --- a/.github/workflows/pytest-dm-tools.yml +++ b/.github/workflows/pytest-dm-tools.yml @@ -29,7 +29,7 @@ jobs: - name: Setup Python uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: - python-version: '3.11' + python-version: '3.12' - name: Setup uv uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 diff --git a/.github/workflows/pytest-inference.yml b/.github/workflows/pytest-inference.yml index 506b902e..1e9b49a7 100644 --- a/.github/workflows/pytest-inference.yml +++ b/.github/workflows/pytest-inference.yml @@ -29,7 +29,7 @@ jobs: - name: Setup Python uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 with: - python-version: '3.11' + python-version: '3.12' - name: Setup uv uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 diff --git a/README.md b/README.md index bdb3c304..a1288ee0 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Whether you are evaluating Azure and NVIDIA as a platform for physical AI, plann ./setup-dev.sh ``` -The setup script installs Python 3.11 via [uv](https://docs.astral.sh/uv/), creates a virtual environment, and installs training dependencies. Follow the [Quickstart Guide](docs/getting-started/quickstart.md) for the full deployment walkthrough. +The setup script installs Python 3.12 via [uv](https://docs.astral.sh/uv/), creates a virtual environment, and installs training dependencies. Follow the [Quickstart Guide](docs/getting-started/quickstart.md) for the full deployment walkthrough. ## Documentation @@ -156,7 +156,7 @@ Agents operate within the same security boundaries, managed identities, and RBAC Prerequisites: -- Python 3.11+ +- Python 3.12+ - Docker with NVIDIA Container Toolkit - Terraform 1.5+ (for infrastructure deployment) - Azure CLI with an active subscription diff --git a/data-management/specifications/viewer-deployment.specification.md b/data-management/specifications/viewer-deployment.specification.md index 210337c3..3bfe1801 100644 --- a/data-management/specifications/viewer-deployment.specification.md +++ b/data-management/specifications/viewer-deployment.specification.md @@ -16,7 +16,7 @@ The frontend serves static assets and proxies `/api/` requests to the backend se | Image | Base | Build Context | |----------|--------------|------------------------------------| -| Backend | Python 3.11 | `data-management/viewer/backend/` | +| Backend | Python 3.12 | `data-management/viewer/backend/` | | Frontend | nginx:alpine | `data-management/viewer/frontend/` | Images are pushed to the Azure Container Registry provisioned by the infrastructure domain. diff --git a/data-management/viewer/README.md b/data-management/viewer/README.md index b3612c38..bd61809a 100644 --- a/data-management/viewer/README.md +++ b/data-management/viewer/README.md @@ -9,7 +9,7 @@ A full-stack application for analyzing and annotating robotic training data from ## Prerequisites -- Python 3.11+ +- Python 3.12+ - Node.js 18+ - npm @@ -21,7 +21,7 @@ A full-stack application for analyzing and annotating robotic training data from cd backend # Create virtual environment (using uv) -uv venv --python 3.11 +uv venv --python 3.12 source .venv/bin/activate # Install dependencies (include 'azure' extra for blob storage support) diff --git a/data-pipeline/arc/README.md b/data-pipeline/arc/README.md index 8e9f03d8..3bcc1343 100644 --- a/data-pipeline/arc/README.md +++ b/data-pipeline/arc/README.md @@ -4,18 +4,18 @@ Kubernetes manifests and configuration for Arc-connected edge data pipeline comp ## 📋 Scope -| Area | Description | -|--------------------|----------------------------------------------------------------| -| K8s manifests | Deployments, services, and config maps for recording workloads | -| RBAC policies | Service accounts and role bindings for edge agents | -| Flux configuration | GitOps sync definitions for automated edge deployment | +| Area | Description | +|--------------------|-----------------------------------------------------------------| +| K8s manifests | Deployments, services, and config maps for recording workloads | +| RBAC policies | Service accounts and role bindings for edge agents | +| Flux configuration | GitOps sync definitions for automated edge deployment | | ACSA manifests | PVC and IngestSubvolume templates for cloud-backed edge storage | ## 📄 ACSA Manifests -| File | Description | -|-------------------------------|-------------------------------------------------------| -| `acsa-pvc.yaml` | ReadWriteMany PVC backed by ACSA `cloud-backed-sc` | -| `acsa-ingest-subvolume.yaml` | IngestSubvolume CRD defining Blob sync policy | +| File | Description | +|------------------------------|----------------------------------------------------| +| `acsa-pvc.yaml` | ReadWriteMany PVC backed by ACSA `cloud-backed-sc` | +| `acsa-ingest-subvolume.yaml` | IngestSubvolume CRD defining Blob sync policy | These templates use `envsubst` variables rendered by `data-pipeline/setup/deploy-acsa.sh`. See the [ACSA setup guide](../../docs/data-pipeline/acsa-setup.md) for full deployment instructions. diff --git a/data-pipeline/setup/README.md b/data-pipeline/setup/README.md index 01b0642e..b491eb91 100644 --- a/data-pipeline/setup/README.md +++ b/data-pipeline/setup/README.md @@ -4,17 +4,17 @@ Deployment scripts for Arc-connected edge agents that run the ROS 2 recording se ## 📋 Scope -| Area | Description | -|-------------------------|---------------------------------------------------------| -| Arc agent provisioning | Connect edge devices to Azure Arc-enabled Kubernetes | -| Connectivity validation | Verify cloud connectivity and service endpoints | -| Runtime configuration | Deploy recording configuration and service dependencies | +| Area | Description | +|-------------------------|-----------------------------------------------------------------| +| Arc agent provisioning | Connect edge devices to Azure Arc-enabled Kubernetes | +| Connectivity validation | Verify cloud connectivity and service endpoints | +| Runtime configuration | Deploy recording configuration and service dependencies | | ACSA deployment | Install Azure Container Storage for Arc and configure Blob sync | ## 📜 Scripts -| Script | Purpose | -|--------------------|-----------------------------------------------------------------------------------| -| `deploy-acsa.sh` | Install cert-manager + ACSA extensions, assign Blob role, apply PVC/subvolume manifests | +| Script | Purpose | +|------------------|-----------------------------------------------------------------------------------------| +| `deploy-acsa.sh` | Install cert-manager + ACSA extensions, assign Blob role, apply PVC/subvolume manifests | See the [ACSA setup guide](../../docs/data-pipeline/acsa-setup.md) for deployment instructions. diff --git a/docs/contributing/README.md b/docs/contributing/README.md index 45eef877..fffd666c 100644 --- a/docs/contributing/README.md +++ b/docs/contributing/README.md @@ -68,7 +68,7 @@ Install required tools and configure Azure access before contributing. See [Prer | kubectl | 1.31 | | Helm | 3.16 | | Node.js | 20+ LTS | -| Python | 3.11+ | +| Python | 3.12+ | | shellcheck | 0.10+ | ## 📜 Code of Conduct diff --git a/docs/data-pipeline/acsa-setup.md b/docs/data-pipeline/acsa-setup.md index 4f547d24..9c79b596 100644 --- a/docs/data-pipeline/acsa-setup.md +++ b/docs/data-pipeline/acsa-setup.md @@ -50,14 +50,14 @@ Recording pods mount the `recording-data` PVC and write bag files to it. The ACS ## 📋 Prerequisites -| Requirement | Details | -|--------------------------------|--------------------------------------------------------------------------------------| -| Azure Arc-connected K8s cluster | Edge cluster registered with Azure Arc (`az connectedk8s show`) | -| Azure CLI 2.60+ | With `k8s-extension` and `connectedk8s` extensions | -| Terraform outputs | Infrastructure deployed via `infrastructure/terraform/` with a storage account | -| kubectl + envsubst | For manifest rendering and application | -| Azure RBAC | Contributor on the Arc cluster resource group; Storage Blob Data Owner on the storage account | -| Network connectivity | Direct kubectl access or Arc proxy for private clusters | +| Requirement | Details | +|---------------------------------|-----------------------------------------------------------------------------------------------| +| Azure Arc-connected K8s cluster | Edge cluster registered with Azure Arc (`az connectedk8s show`) | +| Azure CLI 2.60+ | With `k8s-extension` and `connectedk8s` extensions | +| Terraform outputs | Infrastructure deployed via `infrastructure/terraform/` with a storage account | +| kubectl + envsubst | For manifest rendering and application | +| Azure RBAC | Contributor on the Arc cluster resource group; Storage Blob Data Owner on the storage account | +| Network connectivity | Direct kubectl access or Arc proxy for private clusters | > [!NOTE] > The deploy script automatically installs missing Azure CLI extensions (`k8s-extension`, `connectedk8s`). @@ -86,49 +86,49 @@ The script reads storage account details from `infrastructure/terraform/terrafor ### Script Arguments -| Argument | Environment Variable | Default | Description | -|-------------------------------|-----------------------------------|------------------------------------------|--------------------------------------------| -| `--cluster-name` | `ARC_CLUSTER_NAME` | (required) | Arc-connected cluster name | -| `--cluster-resource-group` | `ARC_RESOURCE_GROUP` | (required) | Resource group of the Arc cluster | -| `-t, --tf-dir` | `DEFAULT_TF_DIR` | `../../infrastructure/terraform` | Terraform directory for output discovery | -| `--storage-account` | `STORAGE_ACCOUNT_NAME` | Auto-discovered from Terraform | Storage account name override | -| `--storage-resource-group` | `STORAGE_ACCOUNT_RESOURCE_GROUP` | Same as cluster resource group | Storage account resource group | -| `--connectivity-mode` | `ACSA_CONNECTIVITY_MODE` | `direct` | `direct` or `proxy` | -| `--proxy-port` | `ACSA_PROXY_PORT` | `47011` | Arc proxy port (proxy mode only) | -| `--config-preview` | — | — | Print configuration and exit | +| Argument | Environment Variable | Default | Description | +|----------------------------|----------------------------------|----------------------------------|------------------------------------------| +| `--cluster-name` | `ARC_CLUSTER_NAME` | (required) | Arc-connected cluster name | +| `--cluster-resource-group` | `ARC_RESOURCE_GROUP` | (required) | Resource group of the Arc cluster | +| `-t, --tf-dir` | `DEFAULT_TF_DIR` | `../../infrastructure/terraform` | Terraform directory for output discovery | +| `--storage-account` | `STORAGE_ACCOUNT_NAME` | Auto-discovered from Terraform | Storage account name override | +| `--storage-resource-group` | `STORAGE_ACCOUNT_RESOURCE_GROUP` | Same as cluster resource group | Storage account resource group | +| `--connectivity-mode` | `ACSA_CONNECTIVITY_MODE` | `direct` | `direct` or `proxy` | +| `--proxy-port` | `ACSA_PROXY_PORT` | `47011` | Arc proxy port (proxy mode only) | +| `--config-preview` | — | — | Print configuration and exit | ### Defaults Configuration Central defaults live in `data-pipeline/setup/defaults.conf`. Override any value via environment variables before running the script. -| Variable | Default | Description | -|--------------------------------|---------------------------|----------------------------------------------| -| `ACSA_EXTENSION_VERSION` | `2.11.2` | ACSA Arc extension version | -| `ACSA_RELEASE_TRAIN` | `stable` | Extension release train | -| `ACSA_DISK_STORAGE_CLASS` | `default,local-path` | Backing disk storage classes | -| `ACSA_PVC_NAME` | `recording-data` | PVC name for recording volume | -| `ACSA_PVC_SIZE` | `50Gi` | PVC storage request | -| `ACSA_STORAGE_CLASS` | `cloud-backed-sc` | ACSA storage class name | -| `BLOB_CONTAINER_NAME` | `datasets` | Target Blob Storage container | -| `SUBVOLUME_NAME` | `recordings` | IngestSubvolume resource name | -| `SUBVOLUME_PATH` | `recordings` | Path prefix within the Blob container | -| `ACSA_INGEST_ORDER` | `oldest-first` | File ingest order (`oldest-first`) | -| `ACSA_INGEST_MIN_DELAY_SEC` | `30` | Minimum delay before ingesting a file | -| `ACSA_EVICTION_ORDER` | `unordered` | Cache eviction order | -| `ACSA_EVICTION_MIN_DELAY_SEC` | `600` | Minimum time (seconds) before evicting cached files | -| `ACSA_ON_DELETE` | `trigger-immediate-ingest`| Behavior when the subvolume is deleted | -| `EDGE_NAMESPACE` | `data-pipeline` | Kubernetes namespace for ACSA resources | +| Variable | Default | Description | +|-------------------------------|----------------------------|-----------------------------------------------------| +| `ACSA_EXTENSION_VERSION` | `2.11.2` | ACSA Arc extension version | +| `ACSA_RELEASE_TRAIN` | `stable` | Extension release train | +| `ACSA_DISK_STORAGE_CLASS` | `default,local-path` | Backing disk storage classes | +| `ACSA_PVC_NAME` | `recording-data` | PVC name for recording volume | +| `ACSA_PVC_SIZE` | `50Gi` | PVC storage request | +| `ACSA_STORAGE_CLASS` | `cloud-backed-sc` | ACSA storage class name | +| `BLOB_CONTAINER_NAME` | `datasets` | Target Blob Storage container | +| `SUBVOLUME_NAME` | `recordings` | IngestSubvolume resource name | +| `SUBVOLUME_PATH` | `recordings` | Path prefix within the Blob container | +| `ACSA_INGEST_ORDER` | `oldest-first` | File ingest order (`oldest-first`) | +| `ACSA_INGEST_MIN_DELAY_SEC` | `30` | Minimum delay before ingesting a file | +| `ACSA_EVICTION_ORDER` | `unordered` | Cache eviction order | +| `ACSA_EVICTION_MIN_DELAY_SEC` | `600` | Minimum time (seconds) before evicting cached files | +| `ACSA_ON_DELETE` | `trigger-immediate-ingest` | Behavior when the subvolume is deleted | +| `EDGE_NAMESPACE` | `data-pipeline` | Kubernetes namespace for ACSA resources | ### Sync Behavior ACSA `IngestSubvolume` controls how files move from edge to cloud: -| Parameter | Default | Behavior | -|------------------|-----------------------------|---------------------------------------------------------| -| Ingest order | `oldest-first` | Oldest files sync first, preserving recording chronology | -| Ingest delay | 30 seconds | Wait before syncing — avoids uploading files still being written | -| Eviction delay | 600 seconds (10 minutes) | Keep cached files locally after upload for re-reads | -| On delete | `trigger-immediate-ingest` | Upload all remaining data immediately when the subvolume is deleted | +| Parameter | Default | Behavior | +|----------------|----------------------------|---------------------------------------------------------------------| +| Ingest order | `oldest-first` | Oldest files sync first, preserving recording chronology | +| Ingest delay | 30 seconds | Wait before syncing — avoids uploading files still being written | +| Eviction delay | 600 seconds (10 minutes) | Keep cached files locally after upload for re-reads | +| On delete | `trigger-immediate-ingest` | Upload all remaining data immediately when the subvolume is deleted | ## 📦 Deployment Steps @@ -353,8 +353,8 @@ lsof -i :47011 ## 📚 Related Documentation -| Resource | Description | -|----------|-------------| -| [Chunking and Compression Configuration](chunking-compression-config.md) | ROS 2 bag chunking and compression settings for edge recording | -| [Azure Container Storage enabled by Azure Arc](https://learn.microsoft.com/azure/azure-arc/container-storage/) | Microsoft documentation for ACSA | -| [IngestSubvolume specification](https://learn.microsoft.com/azure/azure-arc/container-storage/cloud-ingest-edge-volume-configuration) | CRD reference for `IngestSubvolume` | +| Resource | Description | +|---------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------| +| [Chunking and Compression Configuration](chunking-compression-config.md) | ROS 2 bag chunking and compression settings for edge recording | +| [Azure Container Storage enabled by Azure Arc](https://learn.microsoft.com/azure/azure-arc/container-storage/) | Microsoft documentation for ACSA | +| [IngestSubvolume specification](https://learn.microsoft.com/azure/azure-arc/container-storage/cloud-ingest-edge-volume-configuration) | CRD reference for `IngestSubvolume` | diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md index 61507a70..c00522eb 100644 --- a/docs/getting-started/README.md +++ b/docs/getting-started/README.md @@ -41,7 +41,7 @@ Deploy the Physical AI Toolchain and submit your first training job. This hub gu | Azure CLI | ≥2.65.0 | | kubectl | ≥1.31 | | Helm | ≥3.16 | -| Python | ≥3.11 | +| Python | ≥3.12 | Azure subscription with Contributor + User Access Administrator roles, GPU quota for `Standard_NC24ads_A100_v4`, and an NVIDIA NGC account are required. See [Prerequisites](../contributing/prerequisites.md) for full details. diff --git a/docs/recipes/README.md b/docs/recipes/README.md index 6c0ed60d..fe9d94c3 100644 --- a/docs/recipes/README.md +++ b/docs/recipes/README.md @@ -30,7 +30,7 @@ Step-by-step guides that take you from a standing start to a working result. Eac | Recipe | Description | Prerequisites | |---------------------------------------------------------------------------------------|---------------------------------------------------------------------|-------------------------| | [Configuring Edge Data Recording](data-collection/configuring-edge-data-recording.md) | Set up ROS 2 edge recording on Jetson with chunking and compression | Jetson device, ROS 2 | -| [Preparing Datasets for Training](data-collection/preparing-datasets-for-training.md) | Download, inspect, and validate datasets for LeRobot training | Python 3.11+, Azure CLI | +| [Preparing Datasets for Training](data-collection/preparing-datasets-for-training.md) | Download, inspect, and validate datasets for LeRobot training | Python 3.12+, Azure CLI | ## 🔗 Related Documentation diff --git a/docs/recipes/data-collection/preparing-datasets-for-training.md b/docs/recipes/data-collection/preparing-datasets-for-training.md index 99764bf3..7ad3f4c9 100644 --- a/docs/recipes/data-collection/preparing-datasets-for-training.md +++ b/docs/recipes/data-collection/preparing-datasets-for-training.md @@ -9,7 +9,7 @@ Download a dataset from Azure Blob Storage or HuggingFace, inspect its structure | Requirement | Details | |-----------------|------------------------------------------------------------------| -| Python | 3.11+ with `uv` or `pip` | +| Python | 3.12+ with `uv` or `pip` | | Azure CLI | Authenticated (`az login`) — for Azure Blob datasets | | Azure Storage | Storage account with dataset container — for Azure Blob datasets | | HuggingFace CLI | `pip install huggingface-hub` — for HuggingFace datasets | diff --git a/docs/security/README.md b/docs/security/README.md index 7951a55e..c811c376 100644 --- a/docs/security/README.md +++ b/docs/security/README.md @@ -48,12 +48,12 @@ The [security guide](../operations/security-guide.md) documents: Automated security and freshness checks that run on GitHub Actions schedules and publish findings to the Security tab. -| Script | Workflow | Purpose | -|--------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|----------------------------------------------------------------------------------------------| -| [`scripts/security/Test-BinaryFreshness.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-BinaryFreshness.ps1) | `check-binary-integrity.yml` | Verify pinned binary SHA-256 hashes and detect Helm chart version drift (SARIF output) | -| [`scripts/security/Test-DependencyPinning.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-DependencyPinning.ps1) | `dependency-pinning-scan.yml` | Validate that GitHub Actions, Docker images, and package manifests pin exact versions | -| [`scripts/security/Test-SHAStaleness.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-SHAStaleness.ps1) | `sha-staleness-check.yml` | Detect SHA pins that have drifted behind upstream release tags | -| [`scripts/update-chart-hashes.sh`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/update-chart-hashes.sh) | Run manually after chart bumps | Refresh pinned Helm chart versions and SHA-256 hashes in `infrastructure/setup/defaults.conf` | +| Script | Workflow | Purpose | +|-----------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|-----------------------------------------------------------------------------------------------| +| [`scripts/security/Test-BinaryFreshness.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-BinaryFreshness.ps1) | `check-binary-integrity.yml` | Verify pinned binary SHA-256 hashes and detect Helm chart version drift (SARIF output) | +| [`scripts/security/Test-DependencyPinning.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-DependencyPinning.ps1) | `dependency-pinning-scan.yml` | Validate that GitHub Actions, Docker images, and package manifests pin exact versions | +| [`scripts/security/Test-SHAStaleness.ps1`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/security/Test-SHAStaleness.ps1) | `sha-staleness-check.yml` | Detect SHA pins that have drifted behind upstream release tags | +| [`scripts/update-chart-hashes.sh`](https://github.com/microsoft/physical-ai-toolchain/blob/main/scripts/update-chart-hashes.sh) | Run manually after chart bumps | Refresh pinned Helm chart versions and SHA-256 hashes in `infrastructure/setup/defaults.conf` | Each PowerShell script supports a `-SarifFile` parameter for CI integration and a `-ConfigPreview` switch for local dry-run inspection. Run `scripts/update-chart-hashes.sh` locally whenever a pinned Helm chart version is updated so `defaults.conf` stays in sync. diff --git a/docs/security/workflow-permissions.md b/docs/security/workflow-permissions.md index 0a7b9c2d..eb754cc6 100644 --- a/docs/security/workflow-permissions.md +++ b/docs/security/workflow-permissions.md @@ -27,23 +27,23 @@ This document enumerates every job-scoped write permission across `.github/workf The 15 write permissions below are required by the action or CLI invoked in the corresponding job. Each grant is the minimum scope needed. -| Workflow | Job | Permission | Rationale | -|-----------------------------------|--------------------------------|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------| -| `check-binary-integrity.yml` | `check-binary-integrity` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish binary integrity findings to the Security tab. | -| `codeql-analysis.yml` | `analyze` | `security-events: write` | Required by `github/codeql-action/analyze` to upload CodeQL SARIF results to the Security tab. | -| `dast-zap-scan.yml` | `dast-zap-scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish ZAP DAST findings to the Security tab. | -| `dependency-pinning-scan.yml` | `dependency-pinning-scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish SHA-pinning findings to the Security tab. | -| `gitleaks-scan.yml` | `scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish secret-scanning findings to the Security tab. | -| `main.yml` | `dependency-pinning` | `security-events: write` | Inherited by reusable `dependency-pinning-scan.yml`; required for SARIF upload. | -| `main.yml` | `codeql-analysis` | `security-events: write` | Inherited by reusable `codeql-analysis.yml`; required for SARIF upload. | -| `main.yml` | `generate-dependency-sbom` | `contents: write` | Required by `gh release upload "${TAG}" dependencies.spdx.json --clobber` to attach the dependency SBOM to the release. | -| `main.yml` | `attest-release` | `attestations: write` | Required by `actions/attest-build-provenance` and `actions/attest` to create Sigstore provenance attestations. | -| `main.yml` | `attest-release` | `contents: write` | Required by `gh release upload` to attach `*.sigstore.json` and `*.intoto.jsonl` attestation artifacts to the release. | -| `main.yml` | `sbom-diff` | `contents: write` | Required by `gh release upload "${TAG}" dependency-diff.md --clobber` to attach the dependency-change report to the release. | -| `main.yml` | `append-verification-notes` | `contents: write` | Required by `gh release edit` to append artifact-verification instructions to the release body. | -| `pr-validation.yml` | `dependency-pinning` | `security-events: write` | Inherited by reusable `dependency-pinning-scan.yml`; required for SARIF upload. | -| `pr-validation.yml` | `codeql-analysis` | `security-events: write` | Inherited by reusable `codeql-analysis.yml`; required for SARIF upload. | -| `scorecard.yml` | `analysis` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish OpenSSF Scorecard findings to the Security tab. | +| Workflow | Job | Permission | Rationale | +|-------------------------------|-----------------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------| +| `check-binary-integrity.yml` | `check-binary-integrity` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish binary integrity findings to the Security tab. | +| `codeql-analysis.yml` | `analyze` | `security-events: write` | Required by `github/codeql-action/analyze` to upload CodeQL SARIF results to the Security tab. | +| `dast-zap-scan.yml` | `dast-zap-scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish ZAP DAST findings to the Security tab. | +| `dependency-pinning-scan.yml` | `dependency-pinning-scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish SHA-pinning findings to the Security tab. | +| `gitleaks-scan.yml` | `scan` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish secret-scanning findings to the Security tab. | +| `main.yml` | `dependency-pinning` | `security-events: write` | Inherited by reusable `dependency-pinning-scan.yml`; required for SARIF upload. | +| `main.yml` | `codeql-analysis` | `security-events: write` | Inherited by reusable `codeql-analysis.yml`; required for SARIF upload. | +| `main.yml` | `generate-dependency-sbom` | `contents: write` | Required by `gh release upload "${TAG}" dependencies.spdx.json --clobber` to attach the dependency SBOM to the release. | +| `main.yml` | `attest-release` | `attestations: write` | Required by `actions/attest-build-provenance` and `actions/attest` to create Sigstore provenance attestations. | +| `main.yml` | `attest-release` | `contents: write` | Required by `gh release upload` to attach `*.sigstore.json` and `*.intoto.jsonl` attestation artifacts to the release. | +| `main.yml` | `sbom-diff` | `contents: write` | Required by `gh release upload "${TAG}" dependency-diff.md --clobber` to attach the dependency-change report to the release. | +| `main.yml` | `append-verification-notes` | `contents: write` | Required by `gh release edit` to append artifact-verification instructions to the release body. | +| `pr-validation.yml` | `dependency-pinning` | `security-events: write` | Inherited by reusable `dependency-pinning-scan.yml`; required for SARIF upload. | +| `pr-validation.yml` | `codeql-analysis` | `security-events: write` | Inherited by reusable `codeql-analysis.yml`; required for SARIF upload. | +| `scorecard.yml` | `analysis` | `security-events: write` | Required by `github/codeql-action/upload-sarif` to publish OpenSSF Scorecard findings to the Security tab. | ## 🛡️ Defense in Depth diff --git a/scripts/README.md b/scripts/README.md index b7a11535..e636c419 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -66,12 +66,12 @@ PowerShell scripts for validating code quality and documentation. Security scanning and dependency management scripts. -| Script | Purpose | -|--------------------------------|--------------------------------------------------------------------------------------------| -| `security/Test-DependencyPinning.ps1` | Validate dependency pinning compliance | -| `security/Test-SHAStaleness.ps1` | Check for outdated SHA pins | -| `security/Test-BinaryFreshness.ps1` | Validate pinned binary hashes and Helm chart versions; emits SARIF for GitHub Security tab | -| `security/zap-to-sarif.py` | Convert ZAP results to SARIF format | +| Script | Purpose | +|---------------------------------------|-----------------------------------------------------------------------------------------------| +| `security/Test-DependencyPinning.ps1` | Validate dependency pinning compliance | +| `security/Test-SHAStaleness.ps1` | Check for outdated SHA pins | +| `security/Test-BinaryFreshness.ps1` | Validate pinned binary hashes and Helm chart versions; emits SARIF for GitHub Security tab | +| `security/zap-to-sarif.py` | Convert ZAP results to SARIF format | | `update-chart-hashes.sh` | Refresh pinned Helm chart versions and SHA-256 hashes in `infrastructure/setup/defaults.conf` | The `Test-BinaryFreshness.ps1` script is invoked by the `check-binary-integrity.yml` workflow on a weekly schedule. It downloads each pinned GPG key, installer, and CLI archive, compares SHA-256 hashes against the values pinned in `.devcontainer/install-dev-deps.sh` and `.devcontainer/devcontainer.json`, and queries upstream Helm repositories for chart version drift. Findings are written to `binary-freshness-results.sarif` with per-rule `helpUri` values pointing at the appropriate remediation script. @@ -80,19 +80,19 @@ The `Test-BinaryFreshness.ps1` script is invoked by the `check-binary-integrity. Pins are split across two files by structural necessity, not duplication. Each file owns a different class of artifact: -| Artifact class | Canonical location | Why it lives there | -|------------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------| -| Helm chart versions + SHAs | `infrastructure/setup/defaults.conf` | Sourced by runtime shell deploy scripts (`infrastructure/setup/*.sh`); bash-overridable via `.env.local`. | -| Dev container binaries (OSMO CLI, NGC CLI) + SHAs | `.devcontainer/devcontainer.json` | Consumed during Docker image build, before any shell can source bash variables. | +| Artifact class | Canonical location | Why it lives there | +|---------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------| +| Helm chart versions + SHAs | `infrastructure/setup/defaults.conf` | Sourced by runtime shell deploy scripts (`infrastructure/setup/*.sh`); bash-overridable via `.env.local`. | +| Dev container binaries (OSMO CLI, NGC CLI) + SHAs | `.devcontainer/devcontainer.json` | Consumed during Docker image build, before any shell can source bash variables. | All other references to these pins are read-only consumers: -| Consumer | Role | -|-----------------------------------------|------------------------------------------------------------------------------------------| -| `scripts/update-chart-hashes.sh` | Writes chart versions + SHAs back into `defaults.conf` via `sed`; no other file touched. | +| Consumer | Role | +|---------------------------------------------|---------------------------------------------------------------------------------------------------| +| `scripts/update-chart-hashes.sh` | Writes chart versions + SHAs back into `defaults.conf` via `sed`; no other file touched. | | `scripts/security/Test-BinaryFreshness.ps1` | Reads both canonical files (`Get-ShellVariable`, `Get-JsonVariable`) to compare against upstream. | -| `docs/contributing/component-updates.md` | Documents `defaults.conf` as authoritative for chart pins. | -| `.env.local.example` | User-override stubs only — does not redefine defaults. | +| `docs/contributing/component-updates.md` | Documents `defaults.conf` as authoritative for chart pins. | +| `.env.local.example` | User-override stubs only — does not redefine defaults. | ### 🔄 Updating Chart Pins