Skip to content

fix(deployments): docker backend delivers config_files; delete agents docker env bodge - #1216

Closed
maxdubrinsky wants to merge 1 commit into
mainfrom
aircore-955-docker-config-files-delivery/mdubrinsky
Closed

fix(deployments): docker backend delivers config_files; delete agents docker env bodge#1216
maxdubrinsky wants to merge 1 commit into
mainfrom
aircore-955-docker-config-files-delivery/mdubrinsky

Conversation

@maxdubrinsky

Copy link
Copy Markdown
Contributor

Summary

The docker deployments backend ignored DeploymentConfig.config_files, so the agents compiler smuggled the NAT config through a NAT_CONFIG_YAML env var and reconstituted it with a shell one-liner. That bodge made every other substrate inherit docker's config-delivery behavior, breaking OpenShell-backed deployments (nvbugs 6581239). Docker now delivers config_files via create + put_archive + start, the agents mode == "docker" env-smuggle branch is deleted, and config_mount_path moves to /tmp/nemo/config.yaml (the writable intersection of all three substrates).

Related Issue

Part of AIRCORE-955 / AIRCORE-1016. Items 1 (openshell config_files delivery) and 2 (openshell readiness gating) already landed in #1145 and #1139. Item 3 (NAT on PATH inside openshell) is a separable openshell-specific gap tracked separately.

Changes

  • docker/backend.py: new _config_files_tar() builds an in-memory tar (with dir entries) from ConfigFile list. _run_server_container now does containers.create -> put_archive("/", tar) -> container.start(). Config is written into the container's own filesystem before the server command reads it, with no host-path coupling (correct when nmp-api itself runs in a container). Init and sidecar containers move to create + start for a uniform API; the volume-chmod docker run --rm stays on run.
  • runner/deployments_backend.py: deleted the mode == "docker" env-smuggle branch (NAT_CONFIG_YAML / STAGED_CONFIG_FILES_B64_JSON / sh -c materialization). build_deployment_config always emits the direct server command and carries config_files on the DeploymentConfig, matching the k8s path. Removed dead helpers and unused imports (base64, json, shlex).
  • config.py: config_mount_path default /workspace/config.yaml -> /tmp/nemo/config.yaml. /tmp is writable by the docker container user, the openshell sandbox user (uid 999, cannot write /workspace), and k8s ConfigMap subPath (read-only).
  • Tests updated to model containers.create + container.start instead of containers.run.

Type of Change

  • Code change (feature, bug fix, or refactor)

Quality Gates

  • Tests added or updated for changed behavior
  • Documentation not applicable — justification: no user-visible behavior change; internal config-delivery mechanism

Verification

  • Pull request title follows the repository's Conventional Commit format
  • Every commit includes an appropriate Signed-off-by: trailer
  • uv run pre-commit run -a passes, or any blocked checks are identified below
  • Targeted tests pass, or tests are marked not applicable above
  • No secrets, API keys, or credentials are included

Targeted validation:

  • uv run pre-commit run --files <changed paths> — all hooks pass (ruff, ruff format, ty, copyright, merge-conflict).
  • uv run --frozen pytest plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py plugins/nemo-deployments/tests/unit/backends/docker/test_idempotency.py plugins/nemo-deployments/tests/unit/backends/docker/test_image_pull_auth.py plugins/nemo-agents/tests/unit/test_runner_deployments.py -q — 106 passed.
  • uv run --frozen ty check on changed backend files — all checks passed.
  • DCO audit: 1 commit, DCO OK.

Pre-existing failures confirmed identical on origin/main (not caused by this change): port-allocation tests (environment-dependent real port binding), test_cli_list_output (timestamp substring), openshell integration test (AsyncEntitiesResource mock), and test_backend.py basename collection conflict between k8s and openshell dirs.

… docker env bodge (AIRCORE-955)

The docker deployments backend ignored DeploymentConfig.config_files,
so the agents compiler smuggled the NAT config through NAT_CONFIG_YAML
and reconstituted it with a shell one-liner. That bodge inherited
docker's config-delivery behavior on every other substrate, breaking
OpenShell-backed deployments (nvbugs 6581239).

Docker backend now delivers config_files via create + put_archive +
start: the file is written into the container's own filesystem before
the server command reads it, with no host-path coupling (correct when
nmp-api itself runs in a container). Init and sidecar containers move
to create + start for a uniform API; the volume-chmod docker run --rm
stays on run.

The agents mode == docker env-smuggle branch is deleted; build_deployment_config
always emits the direct server command and carries config_files on the
DeploymentConfig, matching the k8s path. config_mount_path default moves
from /workspace/config.yaml to /tmp/nemo/config.yaml: /tmp is the
writable intersection of the docker container user, the openshell
sandbox user (uid 999, cannot write /workspace), and k8s ConfigMap
subPath (read-only).

Items 1 and 2 (openshell config_files delivery and readiness gating)
already landed in #1145 and #1139. Item 3 (NAT on PATH inside
openshell) is a separable openshell-specific gap tracked separately.

Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
@github-actions github-actions Bot added the fix label Aug 10, 2026
@maxdubrinsky

Copy link
Copy Markdown
Contributor Author

Closing to revise commit history; will reopen as a new draft.

@github-actions

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 32006/40616 78.8% 63.6%
Integration Tests 18553/38542 48.1% 20.8%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant