fix(infrastructure): OSMO workflow execution, PostgreSQL public access, and quickstart corrections

[bindsi]

Fixed end-to-end OSMO workflow execution by resolving the empty service_base_url that prevented osmo-ctrl sidecars from connecting to the logger WebSocket. Added PostgreSQL public network access support driven by should_enable_public_network_access, and corrected the quickstart guide with accurate variables, paths, namespaces, and deployment steps validated against a live cluster.

Closes #476
Closes #470

Type of Change

• 🐛 Bug fix (non-breaking change fixing an issue)
• ✨ New feature (non-breaking change adding functionality)
• 💥 Breaking change (fix or feature causing existing functionality to change)
• 📚 Documentation update
• 🏗️ Infrastructure change (Terraform/IaC)
• ♻️ Refactoring (no functional changes)

Component(s) Affected

• infrastructure/terraform/prerequisites/ - Azure subscription setup
• infrastructure/terraform/ - Terraform infrastructure
• infrastructure/setup/ - OSMO control plane / Helm
• workflows/ - Training and evaluation workflows
• training/ - Training pipelines and scripts
• docs/ - Documentation

Testing Performed

• Terraform plan reviewed (no unexpected changes)
• Terraform apply tested in dev environment
• Training scripts tested locally with Isaac Sim
• OSMO workflow submitted successfully
• Smoke tests passed (smoke_test_azure.py)

Validated against live cluster aks-osmo-dev-001 with OSMO v6.0.0 in westus2. Workflow hello-osmo-7 completed successfully after the service_base_url fix. PostgreSQL connectivity verified from AKS pods with public access enabled. All Terraform, shellcheck, and markdownlint validations pass.

Documentation Impact

• No documentation changes needed
• Documentation updated in this PR
• Documentation issue filed

Bug Fix Checklist

• Linked to issue being fixed
• Regression test included, OR
• Justification for no regression test: Infrastructure and deploy script changes validated via live cluster deployment and terraform validate. Quickstart corrections verified by cross-referencing against actual source files.

Description

OSMO Service URL and Workflow Execution

Workflow pods failed silently because the service_base_url in the OSMO SERVICE config was empty, leaving the osmo-ctrl sidecar's -host argument blank and preventing WebSocket connections to the logger service.

• Added detect_ingress_base_url() to scripts/lib/common.sh — returns the AzureML nginx ingress controller in-cluster FQDN (http://azureml-ingress-nginx-controller.azureml.svc.cluster.local) for osmo-ctrl sidecar routing
• Updated 03-deploy-osmo-control-plane.sh to set SERVICE_BASE_URL from the ingress FQDN instead of the external LB IP, ensuring workflow pods route /api/logger, /api/auth, and /api paths correctly
• Added validate_service_url_reachable() to scripts/lib/common.sh — probes the service URL with a 5-second timeout and prints actionable kubectl port-forward guidance when unreachable
• Wired validate_service_url_reachable into both 03-deploy-osmo-control-plane.sh and 04-deploy-osmo-backend.sh
• Split cluster_service_url from CLI URL in 04-deploy-osmo-backend.sh so --service-url http://localhost:8080 works for port-forwarded CLI calls while Helm uses the in-cluster agent_service_url for pods

PostgreSQL Public Network Access

• Changed public_network_access_enabled in infrastructure/terraform/modules/platform/postgresql.tf from hardcoded false to var.should_enable_public_network_access, aligning with other resources (Storage, Redis, Key Vault, ACR)
• Added AllowAzureServices firewall rule (0.0.0.0/0.0.0.0) when public access is enabled, allowing AKS pods to reach PostgreSQL without private endpoints
• Added aks_egress firewall rule scoped to the NAT Gateway public IP when !pe_enabled && nat_gw, for tighter production configurations
• Documented should_enable_public_network_access behavior in both terraform.tfvars and terraform.tfvars.example

Quickstart Guide Corrections

Rewrote docs/getting-started/quickstart.md with corrections validated against the actual codebase:

• Replaced fictional Terraform variables (project_name, enable_*) with actual inputs (resource_prefix, should_*)
• Fixed Terraform output access from -raw resource_group_name to -json resource_group | jq -r '.name'
• Corrected training script path from scripts/submit-osmo-training.sh to training/rl/scripts/submit-osmo-training.sh
• Changed training pod namespace from osmo-control-plane to osmo-workflows
• Updated default GPU SKU to Standard_NV36ads_A10_v5 (A10 Spot)
• Added Step 5: Set NGC API Key and --config-preview dry-run commands
• Added resource_prefix naming constraints warning and Helm cleanup before terraform destroy
• Added devcontainer port-forward guidance for scripts 03 and 04

Troubleshooting and Operations Docs

• Added two entries to docs/operations/troubleshooting.md: "workflow completes but status stays RUNNING" (empty service_base_url) and "UI shows server IP not found" (FQDN vs browser resolution)
• Corrected internal LB IP from 10.0.5.7 to 10.0.5.6 in docs/infrastructure/cluster-setup-advanced.md and fixed the service name reference
• Added [!IMPORTANT] callout explaining service_base_url VPN vs non-VPN behavior

Table Formatting

Ran npm run format:tables across 6 documentation files — column-width normalization only, no content changes.

Related Issues

Closes #476
Closes #470
Subsumes PR #471 by @katriendg

Notes

• The AllowAzureServices PostgreSQL firewall rule is intentionally broad for development scenarios. For production, disable it and rely on the NAT Gateway egress rule or private endpoints.
• The OSMO backend values file (osmo-backend-operator.yaml) was updated from placeholder values to concrete 6.0.0 image tag and in-cluster agent URL. These are overridden by Helm --set-string flags at deploy time.

Checklist

• My code follows the project conventions
• Commit messages follow conventional commit format
• I have performed a self-review
• Documentation impact assessed above
• No new linting warnings introduced

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 94c25a9.

Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.18%. Comparing base (46db49b) to head (94c25a9).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #477   +/-   ##
=======================================
  Coverage   65.18%   65.18%           
=======================================
  Files         253      253           
  Lines       15541    15541           
  Branches     2118     2118           
=======================================
  Hits        10130    10130           
  Misses       5122     5122           
  Partials      289      289           

Flag	Coverage Δ
pester	82.24% <ø> (ø)
pytest	92.40% <ø> (ø)
pytest-dataviewer	65.12% <ø> (ø)
pytest-fuzz	1.56% <ø> (ø)
vitest	51.77% <ø> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[katriendg]

Thanks, looks like everything is there for an initial fix!