feat: unify Docker capability detection (AIRCORE-971)

[tylersbray]

Summary

• Introduce a shared Docker capability probe (nemo_platform_plugin.capabilities) with ProbeResult, per-docker_host caching, and reset_capability_cache for tests.
• Wire jobs merge/registry/init, deployments registry/backends, setup, and quickstart through the shared probe; MissingBackendDependencyError now subclasses CapabilityUnavailableError.
• Gate Automodel/Unsloth compile on probe_docker instead of treating Runtime.DOCKER/NONE as the Docker signal (async compile runs the sync probe via asyncio.to_thread).
• Keep DOCKER→NONE soft-downgrade as a deprecated convenience toward AIRCORE-972.

Depends on the tactical soft-skip in #997. Capability-driven default selection (shrinking/removing Runtime) remains AIRCORE-972.

Linear: AIRCORE-971

Design contract

• Skip optional backends when their capability is missing.
• Fail closed when an explicitly configured deployments default_executor could not register — owned by deployments ExecutorRegistry (behavior from fix: properly handle NONE case for runtime executor #997; this PR clarifies the user-facing message). Models import-time fail-closed is out of scope: models cannot map executor name→Docker backend, and raising in merge_backends at import would kill the whole process. Soft-downgrade already clears auto models docker_executor defaults under NONE.
• Probe cache: import-time jobs merge uses probe_docker(use_cache=False) so it does not pin the process cache. BackendRegistry.from_config is the authoritative cached boot probe; customization compile shares that cached verdict (restart required after starting Docker mid-process). CLI/quickstart use use_cache=False for retry UX.
• GET /v2/execution-profiles returns 503 until the jobs controller registry has constructed and pruned advertised profiles.
• Independent API vs controller probes in split topologies remain a known limitation (no snapshot API in this PR).

Runtime inventory (AIRCORE-972 handoff)

Topology (keep for shrink decision): Runtime enum + soft-downgrade; jobs/models default profile/backend selection; models executor_for_runtime / deployments_enabled; compiler paths; customization multi-node + require_distributed_runtime.

Capability (migrated here): jobs merge/registry Docker filter; deployments Docker backend/registry; setup/quickstart; customization require_container_runtime.

Deferred: agents improvement/preflight.py (docker info); deployments docker/gpu.py pool recovery; API/controller capability snapshot; GPU/k8s probe extension points (stubs intentionally not shipped).

Test plan

• uv run --frozen pytest packages/nemo_platform_plugin/tests/test_capabilities.py
• Full plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py (includes init-unavailable cases; suite lacks unit marker — run explicitly)
• uv run --frozen pytest services/core/jobs/tests/test_config.py plugins/nemo-deployments/tests/unit/test_registry.py
• uv run --frozen pytest packages/nmp_customization_common/tests/contributor/test_jobs.py packages/nmp_common/tests/jobs/test_docker.py
• GET /v2/execution-profiles 503-until-ready unit coverage
• make vendor-nemo-platform-ext after ext changes
• uv.lock vs main is +2 lines only (nmp-jobs → nemo-platform-plugin)
• Full local.yaml with Docker down → expect fail-closed at deployments default_executor: local-docker
• Reduced no-Docker config → platform boots; no docker job profiles after controller ready
• Full local.yaml with Docker up → docker profiles + deployments default register

Summary by CodeRabbit

• New Features
  • Added centralized Docker availability detection with detailed results, caching, timeout handling, and configurable hosts.
  • Improved Docker validation across job execution, deployments, setup, and quickstart workflows.
  • Added readiness gating for execution-profile APIs, returning HTTP 503 until backends initialize.
  • GPU validation now accounts for Docker availability and runtime configuration.
• Bug Fixes
  • Prevented Docker checks from blocking asynchronous job compilation.
  • Preserved Docker TLS settings for configured hosts.
  • Improved handling of unavailable execution backends and startup failures.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Docker capability migration

Layer / File(s)	Summary
Capability probe implementation packages/nemo_platform_plugin/src/nemo_platform_plugin/capabilities.py, packages/nemo_platform_plugin/tests/test_capabilities.py	Adds structured Docker probe results, caching, timeout handling, cleanup, cache reset support, and require_docker.
Platform and runtime consumers packages/nemo_platform_ext/..., packages/nemo_platform_plugin/..., packages/nmp_common/..., packages/nmp_customization_common/..., plugins/nemo-automodel/..., plugins/nemo-unsloth/...	Replaces direct Docker SDK checks with cached or uncached capability probes. Runtime and GPU validation use probe results and details.
Deployment backend integration plugins/nemo-deployments/...	Docker backend initialization probes the configured host and raises the shared unavailable-capability error hierarchy.
Core job backend integration services/core/jobs/..., services/core/jobs/pyproject.toml, packages/nemo_platform/pyproject.toml	Job backend selection and initialization use capability probes. Workspace dependencies are declared.
Execution-profile readiness gate services/core/jobs/src/nmp/core/jobs/config.py, services/core/jobs/src/nmp/core/jobs/api/v2/jobs/endpoints.py, services/core/jobs/tests/test_jobs_client.py	The execution-profiles endpoint returns HTTP 503 until backend registration completes.

Sequence Diagram(s)

sequenceDiagram
  participant JobsAPI
  participant BackendRegistry
  participant probe_docker
  participant DockerDaemon
  JobsAPI->>BackendRegistry: request execution profiles
  BackendRegistry->>probe_docker: probe Docker availability
  probe_docker->>DockerDaemon: create client and ping
  DockerDaemon-->>probe_docker: availability and detail
  probe_docker-->>BackendRegistry: ProbeResult
  BackendRegistry->>BackendRegistry: register backends and mark profiles ready
  BackendRegistry-->>JobsAPI: readiness state
  JobsAPI-->>JobsAPI: return profiles or HTTP 503

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#997: Modifies Docker and runtime handling across related validation and backend paths.
• NVIDIA-NeMo/nemo-platform#684: Modifies container-runtime validation in the contributor job path.
• NVIDIA-NeMo/nemo-platform#766: Modifies Docker backend availability handling for deployment flows.

Suggested reviewers: svvarom, mckornfield

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: unifying Docker capability detection across the affected components.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch AIRCORE-971-unify-capability-detection/tbray

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/capabilities.py`:
- Around line 119-123: The Docker host is passed using unsupported base_url
kwargs. In capabilities.py, update both Docker client initialization sites to
copy the environment, set DOCKER_HOST when docker_host is configured, and pass
that environment to docker.from_env instead of base_url; update
test_capabilities.py to assert kwargs["environment"]["DOCKER_HOST"] for the
configured host case.

In `@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/registry.py`:
- Around line 84-86: Update the error guidance in the backend
capability/configuration message to be backend-neutral instead of directing
operators to configure a non-Docker executor or reach Docker. Derive recovery
advice from the matching ExecutorSpec when available, or use generic guidance
that applies to any executor backend, while preserving the existing error
context.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9a46a7b4-6919-44b6-9201-65c032296af2

📥 Commits

Reviewing files that changed from the base of the PR and between f45c965 and 74e9052.

⛔ Files ignored due to path filters (5)

• sdk/python/nemo-platform/src/nemo_platform/cli/commands/setup.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/quickstart/preflight.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/quickstart/validators.py is excluded by !sdk/**
• sdk/python/nemo-platform/tests/vendored/nemo_platform_ext/cli/commands/test_setup.py is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (18)

• packages/nemo_platform_ext/src/nemo_platform_ext/cli/commands/setup.py
• packages/nemo_platform_ext/src/nemo_platform_ext/quickstart/preflight.py
• packages/nemo_platform_ext/src/nemo_platform_ext/quickstart/validators.py
• packages/nemo_platform_ext/tests/cli/commands/test_setup.py
• packages/nemo_platform_plugin/src/nemo_platform_plugin/capabilities.py
• packages/nemo_platform_plugin/src/nemo_platform_plugin/config.py
• packages/nemo_platform_plugin/tests/test_capabilities.py
• packages/nemo_platform_plugin/tests/test_config.py
• packages/nmp_customization_common/src/nmp/customization_common/contributor/jobs.py
• packages/nmp_customization_common/tests/contributor/test_jobs.py
• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/base.py
• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/registry.py
• services/core/jobs/pyproject.toml
• services/core/jobs/src/nmp/core/jobs/controllers/backends/config.py
• services/core/jobs/src/nmp/core/jobs/controllers/backends/docker.py
• services/core/jobs/src/nmp/core/jobs/controllers/backends/registry.py
• services/core/jobs/tests/test_config.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31886/40497	78.7%	63.4%
Integration Tests	18513/38423	48.2%	20.8%

[tylersbray]

@coderabbitai resume
@coderabbitai review

[coderabbitai]

@tylersbray Review resumed. Starting review.

✅ Action performed

Reviews resumed.

---

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[mckornfield]

some q

[ironcommit]

apart from the global state, a step in the right direction. This start up code still needs more work.

[tylersbray]

Update after plan-audit + product sync

Rebased onto current main and pushed two follow-up commits:

1. 54530a4e5d — remove the controller-only _execution_profiles_ready / 503 gate (split-topology bug + Ryan’s global-state review). Address Matt/Ryan review threads (docker env helper, automodel casts, probe fixture, jobs→plugin dep rationale).
2. 45fe6ac811 — config-aware Docker preflight for nemo setup + services run/start/restart (NVBug 6537617). Triggers only when resolved selection starts deployments with a docker-backed default_executor (pre-soft-downgrade YAML intent). Not a global “Docker mandatory for NeMo” gate; kubernetes / reduced selections still boot. Restart preflights before stop.

Accepted limitation this release: Harbor/evals on already-booted reduced configs can still fail at use-time.

Linear: AIRCORE-971 (direction comment posted).

[tylersbray]

/nvskills-ci