feat(infrastructure): upgrade OSMO to chart 1.2.1 / image 6.2 with secure auth and skrl 2.0.0 compatibility

[katriendg]

Description

Upgraded the OSMO platform from chart 1.0.1 (image 6.0.0) to chart 1.2.1 (image 6.2) and replaced the broken dev-mode authentication with token-based auth backed by a Key Vault credential lifecycle. The previous deployment had three compounding failures — backend CrashLoopBackOff from invalid loginMethod, 404 on all auth APIs, and silently ignored defaultAdmin values — making the platform inoperable.

This PR also absorbed the skrl 2.0.0 breaking API change (private _update → public update with keyword-only arguments) and corrected several training dependency pins that caused import failures in GPU containers. End-to-end training validated on both OSMO and AzureML.

Closes #478

Type of Change

• 🐛 Bug fix (non-breaking change fixing an issue)
• ✨ New feature (non-breaking change adding functionality)
• 💥 Breaking change (fix or feature causing existing functionality to change)
• 📚 Documentation update
• 🏗️ Infrastructure change (Terraform/IaC)
• ♻️ Refactoring (no functional changes)

Component(s) Affected

• infrastructure/terraform/prerequisites/ - Azure subscription setup
• infrastructure/terraform/ - Terraform infrastructure
• infrastructure/setup/ - OSMO control plane / Helm
• workflows/ - Training and evaluation workflows
• training/ - Training pipelines and scripts
• docs/ - Documentation

Testing Performed

• Terraform plan reviewed (no unexpected changes)
• Terraform apply tested in dev environment
• Training scripts tested locally with Isaac Sim
• OSMO workflow submitted successfully
• Smoke tests passed (smoke_test_azure.py)

Validated Training Runs

Run	Platform	Status	Throughput
isaaclab-inline-training-23	OSMO (workload identity)	COMPLETED	8.3 it/s
isaaclab-inline-training-24	OSMO (workload identity + dependabot bumps)	COMPLETED	—
redacted	AzureML K8s compute	SUBMITTED	—

Validation Commands Run

• ruff check training/ evaluation/ — passed
• pytest training/ evaluation/ — 43 passed, 92.5% coverage

Documentation Impact

• No documentation changes needed
• Documentation updated in this PR
• Documentation issue filed

Bug Fix Checklist

• Linked to issue being fixed
• Regression test included, OR
• Justification for no regression test: OSMO deployment and skrl training validated through live GPU runs on the target cluster. Unit tests cover skrl wrapper logic. Infrastructure changes are inherently integration-tested through the deploy + train cycle.

Changes

OSMO 6.2 Platform Upgrade

The Helm chart and image versions were bumped across all four releases (control plane, backend operator, router, UI). The control plane values introduced a defaultAdmin block — NVIDIA's bootstrap mechanism for non-IdP deployments — and removed auth.enabled: false, which was disabling all user/token CRUD endpoints.

• Updated defaults.conf with chart 1.2.1, image 6.2, and CLI version 6.2.10
• Added defaultAdmin configuration and Entra IdP placeholder comments in osmo-control-plane.yaml
• Bumped image tag to 6.2 in osmo-backend-operator.yaml, osmo-router.yaml, and osmo-ui.yaml

Authentication and Credential Management

The old --method dev --username guest login path was removed in OSMO 6.x. Admin credentials now follow the same Key Vault lifecycle as the PostgreSQL password: Terraform generates → Key Vault stores → CSI syncs to K8s secret.

• Added random_password.osmo_admin (43-char alphanumeric, URL-safe for Bearer tokens) and azapi_resource for Key Vault storage in security.tf
• Added osmo-admin-password to both CSI SecretProviderClass manifests for automatic K8s secret sync
• Rewrote osmo_login_and_setup in common.sh to use --method token --token-file with process substitution
• Switched 04-deploy-osmo-backend.sh from raw curl POST to osmo token set CLI with 90-day expiry (was 1 year) and --from-file secret creation to avoid credential exposure in process args
• Added service_base_url self-healing in script 04 — detects when script 03 couldn't set it (private cluster, no port-forward) and applies the in-cluster ingress URL so workflow pod sidecars can reach the control plane

skrl 2.0.0 API Migration

The upstream skrl library promoted the agent update method from a private to a public interface with keyword-only arguments.

• Changed protocol definition and wrapper in skrl_mlflow_agent.py from _update(timestep, timesteps) to update(*, timestep, timesteps)
• Updated monkey-patch assignment in skrl_training.py
• Migrated eval mode in policy_evaluation.py from set_running_mode("eval") to enable_training_mode(enabled=False, apply_to_models=True) and added required second positional argument to act()

Training Dependencies

• Pinned numpy to >=1.26.0,<2.0.0 in pyproject.toml — Isaac Sim 4.x bundles numpy 1.x-compiled native extensions, and numpy 2.x changes the C ABI
• Downgraded marshmallow from 4.x to 3.25.1 — azure-ai-ml requires marshmallow>=3.5,<4.0.0
• Aligned pydantic 2.13.1 / pydantic-core 2.46.1 — mismatched versions raise SystemError on import
• Applied 9 transitive dependency bumps from dependabot PR security(deps): bump the training-dependencies group in /training/rl with 9 updates #490
• Replaced runtime uv pip compile in train.sh with uv pip install --no-deps from the pre-locked requirements.txt, removing non-deterministic resolution during GPU-billed training startup
• Made --max_iterations conditional in train.yaml to prevent empty-string argparse failures

Notes

• Known limitation: osmo workflow logs returns 500 when shared_access_key_enabled = false on the storage account. The OSMO service internally calls BlobServiceClient.from_connection_string() which does not support DefaultAzureCredential. Upstream fix tracked in NVIDIA/OSMO PR #865. Workaround: kubectl logs for live output, az storage blob download for completed runs. Tracked in fix: OSMO workflow log retrieval fails with Azure workload identity #491
• Dataset access-keys template gained an endpoint field in dataset-config-access-keys.template.json

Checklist

• My code follows the project conventions
• Commit messages follow conventional commit format
• I have performed a self-review
• Documentation impact assessed above
• No new linting warnings introduced

[github-actions]

Dependency Review

The following issues were found:

• ✅ 0 vulnerable package(s)
• ✅ 0 package(s) with incompatible licenses
• ✅ 0 package(s) with invalid SPDX license definitions
• ⚠️ 9 package(s) with unknown licenses.

See the Details below.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 160a889.

Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

License Issues

training/rl/pyproject.toml

Package	Version	License	Issue Type
packaging	26.1	Null	Unknown License

training/rl/requirements.txt

Package	Version	License	Issue Type
cuda-pathfinder	1.5.3	Null	Unknown License
filelock	3.28.0	Null	Unknown License
huey	3.0.0	Null	Unknown License
mako	1.3.11	Null	Unknown License
nvidia-cudnn-cu12	9.21.0.82	Null	Unknown License
packaging	26.1	Null	Unknown License
pydantic	2.13.1	Null	Unknown License
zipp	3.23.1	Null	Unknown License

OpenSSF Scorecard

Scorecard details

Package	Version	Score	Details
pip/marshmallow	3.26.2	Unknown	Unknown
pip/numpy	1.26.4	Unknown	Unknown
pip/packaging	26.1	Unknown	Unknown
pip/cuda-pathfinder	1.5.3	Unknown	Unknown
pip/filelock	3.28.0	Unknown	Unknown
pip/huey	3.0.0	Unknown	Unknown
pip/mako	1.3.11	Unknown	Unknown
pip/marshmallow	3.26.2	Unknown	Unknown
pip/numpy	1.26.4	Unknown	Unknown
pip/nvidia-cudnn-cu12	9.21.0.82	Unknown	Unknown
pip/packaging	26.1	Unknown	Unknown
pip/pydantic	2.13.1	Unknown	Unknown
pip/pydantic-core	2.46.1	🟢 6.7	Details Check Score Reason Code-Review 🟢 10 all changesets reviewed Maintained 🟢 10 30 commit(s) and 16 issue activity found in the last 90 days -- score normalized to 10 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Binary-Artifacts 🟢 10 no binaries found in the repo License 🟢 10 license file detected Pinned-Dependencies 🟢 8 dependency not pinned by hash detected -- score normalized to 8 Fuzzing 🟢 10 project is fuzzed Signed-Releases ⚠️ 0 Project has not signed or included provenance with any releases. Branch-Protection 🟢 4 branch protection is not maximal on development and all release branches Security-Policy 🟢 10 security policy file detected Packaging 🟢 10 packaging workflow detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/zipp	3.23.1	Unknown	Unknown

Scanned Files

• training/rl/pyproject.toml
• training/rl/requirements.txt

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.19%. Comparing base (7196d6d) to head (160a889).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #492   +/-   ##
=======================================
  Coverage   65.19%   65.19%           
=======================================
  Files         254      254           
  Lines       15804    15804           
  Branches     2118     2070   -48     
=======================================
  Hits        10303    10303           
  Misses       5212     5212           
  Partials      289      289           

Flag	Coverage Δ
pester	81.11% <ø> (ø)
pytest	92.40% <ø> (ø)
pytest-dataviewer	65.12% <ø> (ø)
pytest-fuzz	1.56% <ø> (ø)
vitest	51.77% <ø> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bindsi]

Review item RI-1: Stale docstring references and return type annotation mismatch after skrl 2.0.0 migration.

[bindsi]

RI-2: Unexplained positional argument in agent.act() call after skrl 2.0.0 migration.

[bindsi]

RI-3 through RI-5 (RI-4 dropped — postgresql.tf firewall rules are from the already-merged PR #477, not this PR).

RI-6 (PR description): The "OSMO CLI Pinning" section states:

Replaced unpinned curl | bash installation with versioned download of OSMO CLI 6.2.10 verified via SHA256 checksum in devcontainer.json

But the devcontainer.json diff in this PR only shows a tflint install hardening (from the already-merged #465). The OSMO CLI version is pinned in defaults.conf via OSMO_CLI_VERSION, which is a different mechanism. Consider correcting the description to avoid confusion during future audits.

[bindsi]

Great work...the comments to optimise are more less low prio findings

[liupeirong]

setup\cleanup\uninstall-osmo-control-plane.sh drops a list of psql tables. Those tables changed from 6.0 to 6.2. Can we perhaps drop all the tables in the osmo DB rather than updating the list for each release.

[katriendg]

setup\cleanup\uninstall-osmo-control-plane.sh drops a list of psql tables. Those tables changed from 6.0 to 6.2. Can we perhaps drop all the tables in the osmo DB rather than updating the list for each release.

Thanks for the review @liupeirong - update the script: replaced the hand-maintained 25-table list with a DROP SCHEMA public CASCADE; CREATE SCHEMA public; approach.
This drops every table, view, sequence, type, and function in one shot, so future OSMO schema changes (6.2 → 6.x) no longer require editing this script.