fix(trtllm): restore deps, default CMD, and reduce layer count after upstream base switch

[tanmayv25]

Three regressions from PR #9654 (TRT-LLM upstream base switch).

1. Restore missing system packages (openssh-server, librdmacm1, rdma-core)

#9654 dropped Dynamo's explicit apt block on the assumption upstream nvcr.io/nvidia/tensorrt-llm/release had everything we need. Three packages turned out to be missing:

• openssh-server — upstream ships openssh-client but no sshd. Breaks multi-node MPI-over-SSH (kimi-k2.5 K8s benchmark).
• librdmacm1 + rdma-core — RDMA Connection Manager userspace; needed for NCCL IB transport and UCX rdmacm mode.

Verified empirically against tensorrt-llm/release:1.3.0rc14. Everything else from the pre-#9654 apt block (build-essential, python3.12-dev, python3-pip, jq, numactl, libibverbs1, libnuma1, libcudnn9-cuda-13, libnvshmem3-cuda-13, libzmq3-dev, openssh-client, curl, git, etc.) is correctly provided by upstream and intentionally not re-added.

After local rebuild: sshd is on PATH at /usr/sbin/sshd, host keys generated by postinst, librdmacm.so.1 loadable via ldconfig, rdma-core package installed.

2. Set default CMD ["/bin/bash"]

#9654 cleared upstream's ENTRYPOINT (/opt/nvidia/nvidia_entrypoint.sh) to let derived runtimes execute arbitrary commands directly, but did not set a CMD. Upstream tensorrt-llm/release has no CMD either, so bare docker run <image> now fails with Error response from daemon: no command specified — breaking downstream pipelines that smoke-test the image without passing an explicit command.

K8s deploy manifests are unaffected: they set command: and args: explicitly, which override CMD.

3. Reduce layer count to clear overlay2's 128-layer cap

Downstream wrapper images (tensorrt-llm_dynamo_trtllm-runtime-with-k8s-wrapper-arm64-benchmarks) were hitting docker: failed to register layer: max depth exceeded at pull time because the runtime image (~111 layers post-#9654) plus the wrapper's additions (~20 layers) plus the upstream PyTorch base (101 layers underneath) was sitting near or above overlay2's 128-layer cap on some hosts.

Architecture: runtime_full + rebase split

The template now produces two different stage chains depending on the build target:

• target=runtime (production image): a layered runtime_full stage does all the build work (apt install, ldconfig, wheels, workspace_src cp), then a final FROM ${RUNTIME_IMAGE} AS runtime stage rebases by COPY --from=runtime_full / / overlaying the entire filesystem as a single layer. Upstream image config (ENV/WORKDIR/USER/CMD — ~40 NVIDIA/CUDA/MPI vars) inherits automatically via FROM upstream; only Dynamo-specific env (DYNAMO_HOME, LD_PRELOAD, NIXL_PLUGIN_DIR, VIRTUAL_ENV, PATH) is redeclared. A RUN rm -rf /workspace /home/ubuntu /usr/local/bin/etcd before the COPY whiteouts upstream paths that runtime_full deleted (COPY can't represent deletions).

• target=dev / target=local-dev: the original layered runtime stage stays as-is; dev/dynamo_tools build directly FROM runtime for fast iteration. No rebase. Wheelhouse COPY is unconditional so tests/dependencies/test_kvbm_imports.py finds the wheels.

Net effect: runtime image drops from 108 → 104 fs layers, with upstream env inheritance preserved (no redeclaration of OPAL_PREFIX, CUDA_HOME, NVIDIA_REQUIRE_CUDA, ~37 other vars).

In-Dockerfile cleanup also included: collapsed transport stages (workspace_files, dynamo_base_export) for one-COPY workspace + dynamo_base imports, consolidated metadata + setup RUNs, restored unconditional wheel COPY (fix for dev/local-dev wheelhouse regression), assert libstdc++.so.6 target exists before symlinking, factor LIBSTDCPP path into a variable.

Test plan

• Standard CI builds the trtllm container image successfully.
• Single-node trtllm smoke tests pass (no regression from the added apt layer).
• docker run nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:<tag> no longer errors with "no command specified" — drops into bash.
• Downstream pipeline test_distroless stage no longer fails with max depth exceeded.
• kimi-k2.5 disagg-eagle-kv-router K8s benchmark on GB200 cluster comes back green (multi-node MPI ranks come up).
• Confirm in container: which sshd returns /usr/sbin/sshd, ldconfig -p | grep librdmacm shows the lib, dpkg -s rdma-core shows installed, /workspace/{tests,examples,deploy,lib,components/src/dynamo/{common,frontend,trtllm,mocker}} all present and owned by dynamo:0.

Layer-count fix verification (new for §3)

• docker inspect dynamo:<runtime-tag> --format '{{len .RootFS.Layers}}' returns ≤104 on amd64.
• Same check on arm64 (build via --platform=linux/arm64).
• docker run --rm <runtime-tag> env | grep -E '^(OPAL_PREFIX|CUDA_HOME|NVIDIA_REQUIRE_CUDA|LD_LIBRARY_PATH)=' returns upstream's values (inheritance via FROM works).
• Disagg smoke test on the rebased runtime image (run_trtllm_disagg.sh with Qwen3-0.6B): POST /v1/chat/completions returns 200; prefill→decode handoff fires with matching request_id.
• docker run --rm <runtime-tag> ls /workspace shows only Dynamo content (no upstream tensorrt_llm/ leak from overlay shadowing).
• docker run --rm <runtime-tag> ls /opt/dynamo/wheelhouse/ shows kvbm, ai_dynamo_runtime, ai_dynamo, gpu_memory_service wheels.
• --target=dev and --target=local-dev builds complete successfully (use the layered runtime stage, not the rebased one — dev iteration speed must remain unaffected).
• --target=dev arm64 build also succeeds (QEMU emulation OK for verification).
• tests/dependencies/test_kvbm_imports.py::test_kvbm_wheel_exists_trtllm passes on the dev test image (wheelhouse populated).

🤖 Generated with Claude Code

[coderabbitai]

Walkthrough

This PR adds openssh-server and RDMA-related OS packages (librdmacm1, rdma-core) to the TensorRT-LLM runtime Docker image. The installation uses cached apt-get with cleanup to minimize the resulting image size.

Changes

Docker Runtime Packages

Layer / File(s)	Summary
SSH and RDMA package installation container/templates/trtllm_runtime.Dockerfile	A new RUN step installs openssh-server and RDMA libraries using apt-get with mounted cache and cleanup of package metadata to reduce image size.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the main change: restoring missing dependencies (openssh-server and rdma-core) in the TRT-LLM runtime image after an upstream base switch.
Description check	✅ Passed	The pull request description comprehensively covers all required template sections with clear rationale, implementation details, verification steps, and test plan.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[dmitry-tokarev-nv]

  Issues / Suggestions
  
  1. Apt cache mount only covers /var/cache/apt, missing /var/lib/apt

  The new RUN block (diff lines for the apt install):

  RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
      apt-get update && ...

  /var/cache/apt caches downloaded .deb files; the package index lives at /var/lib/apt/lists and is re-downloaded by apt-get update every build.
  Adding a second mount:

  RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
      --mount=type=cache,target=/var/lib/apt,sharing=locked \
      apt-get update && ...

  avoids the index re-download on iterative builds (and you can keep the rm -rf /var/lib/apt/lists/* at the end since it operates on a cache-mounted
  path that doesn't persist into the layer anyway). Minor optimization, but free.

  2. librdmacm1 is a transitive dep of rdma-core

  apt-cache depends rdma-core on Ubuntu 24.04 (the upstream TRT-LLM 1.3.x base) lists librdmacm1 as a dependency. Specifying both is harmless but
  redundant — could drop librdmacm1 from the explicit list. Pedantic; leave it if you prefer the explicit documentation.

  3. COPY --chmod=775 dev /dev in scratch stage is technically fine but visually loaded

  /dev in a FROM scratch is just a regular dirname (scratch has no /dev devfs convention), but a reader skimming the Dockerfile will pause on it.
  Consider staging under a non-special prefix:

  FROM scratch AS workspace_files
  COPY --chmod=775 tests /workspace_src/tests
  ...
  COPY --chmod=775 dev /workspace_src/dev

  and copying with COPY --from=workspace_files /workspace_src/ /workspace/. Tradeoff is one extra path component for clarity. Style call.

  4. Inline {% if %} at end of RUN-block continuation
  
      && echo 'umask 002' > /etc/profile.d/00-umask.sh{% if target not in ("dev", "local-dev") %} \
      && python3 -m venv --system-site-packages /opt/dynamo/venv \
      && ln -sf /usr/bin/uv /opt/dynamo/venv/bin/uv{% endif %}

  Mixing Jinja conditionals at the very end of a shell-continuation line is harder to read than the original "separate {% if %} RUN ... {% endif %}
  block". Functionally identical — but the original separation made it obvious that venv creation is a non-dev-only step. Maybe worth keeping the venv
   RUN as its own conditional block (one extra layer for non-dev only, but improves grep-ability).
  
  5. COPY --from=workspace_files / /workspace/ is implicit-include-by-default

  Any future addition to the workspace_files scratch stage will land in /workspace/ of the runtime, whether intended or not. Mostly fine, but worth a
  one-line comment in the scratch stage to flag it ("everything copied into this stage ends up in /workspace/ — keep paths minimal"). Adjacent to the
  existing transport-stage comment.
  
  6. Verify dev/local-dev still builds

  The workspace_files and dynamo_base_export scratch stages run for every target including dev/local-dev. They will build, but the runtime stage's
  flow has been rearranged: the workspace COPY now happens at a fixed point regardless of target. Worth a manual build check for target=dev and
  target=local-dev to confirm no breakage (the test plan covers standard CI but not dev images explicitly).
  
  Security

  - openssh-server in a container image is non-trivial. It installs /usr/sbin/sshd and (per the PR description) generates host keys via postinst. This
   is intentional and matches the multi-node MPI use case, but it does introduce a daemon binary that some org-level container scanners flag (e.g., a
  future SBOM/CVE scan on the runtime image will now also scan openssh server-side bits). Not a blocker — call it out so reviewers don't get surprised
   later by scanner output. Upstream tensorrt-llm/release deliberately ships only openssh-client; we're re-broadening attack surface here in exchange
  for MPI capability.
  - librdmacm1 / rdma-core are RDMA userland — no new exposure for normal workloads.

  Performance

  - Net −7 runtime layers / −4 test_distroless layers per the PR description, which matches my read of the diff (9 workspace COPYs → 1, 3 dynamo_base
  COPYs → 1, 3 prep RUNs → 1, two USER toggles eliminated). The arithmetic checks out.
  - Apt cache mount is good; adding /var/lib/apt cache (point #1) would close the loop.

[dynamo-ops]

Previous review comments have been addressed. Approving.

[dmitry-tokarev-nv]

Inline notes on the new commits — see the separate top-level comment for non-line-anchored concerns.

[dmitry-tokarev-nv]

Remaining concerns from the new commits (not line-anchored)

Follow-up on the four new commits since the last pass (b9451f1, 5f049be, a8087272, 10188fbf). Line-anchored items are in the inline review above; the rest are below.

Concerns addressed since prior review

• ✅ Apt cache /var/lib/apt added — b9451f1.
• ✅ /dev scratch path renamed to /workspace_src/dev — b9451f1.
• ✅ ARG/ENV DYNAMO_COMMIT_SHA moved to bottom for downstream layer-cache reuse — b9451f1. Good call, intentional +1 layer for a real hit-rate improvement.
• ✅ Wheel and workspace COPYs converted to bind-mount + cp inside RUNs — 5f049be. Saves two layers.

Outstanding non-line concerns

1. FROM scratch loses image config that COPY / / doesn't restore

COPY --from=runtime_full / / brings only the filesystem. Upstream's LABEL, HEALTHCHECK, EXPOSE, and STOPSIGNAL are dropped silently. For nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc14:

• LABELs (e.g. org.opencontainers.image.*, com.nvidia.*) — used by registries, SBOM scanners, and downstream tooling that keys off OCI annotations. Worth either re-declaring or explicitly noting in the comment that label propagation is out of scope.
• HEALTHCHECK / EXPOSE / STOPSIGNAL — usually absent on the upstream base, but worth verifying with docker inspect nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc14 to be sure.

2. 10188fbf "Add envvar" is too thin for the change size

The commit adds ~50 environment variables across CUDA / MPI / Triton / PyTorch / Python categories — and has an empty body. For a change of this size:

• Body should describe the capture methodology (docker run --rm <upstream> env), what was filtered/kept, and what verification was performed (ideally a docker inspect runtime_full vs squashed runtime env-parity diff).
• Either fold this into the squash commit (a8087272) or rewrite the message.
• The PR description should also call out the env reconstitution — readers reaching this PR in the future won't grok the squash architecture without it.

3. Wheel persistence — behavior change for dev/local-dev targets

Pre-PR: COPY --from=wheel_builder /opt/dynamo/dist/*.whl /opt/dynamo/wheelhouse/ was unconditional, so dev/local-dev images had a populated wheelhouse.

Post-PR (5f049be): the cp /tmp/wheels/*.whl /opt/dynamo/wheelhouse/ step is inside {% if target not in ("dev", "local-dev") %}. Dev/local-dev images no longer have /opt/dynamo/wheelhouse/*.whl populated.

If tests/dependencies/test_kvbm_imports.py (confirmed file exists) only runs against runtime images, this is fine. If it ever runs against dev images (interactive developer testing, ad-hoc CI), it would start failing. Please confirm the test's runtime context — or, if it should also run against dev, lift the wheel cp out of the conditional.

4. Dev/local-dev build verification not in the test plan

The test plan exercises the standard runtime path. The new conditional structure has meaningfully diverged behavior between target=runtime (squashed, large hardcoded ENV block) and target=dev|local-dev (layered, smaller env). Please add explicit checkboxes to the test plan:

• target=dev builds successfully
• target=local-dev builds successfully
• Verify target=runtime arm64 build (relates to LD_LIBRARY_PATH inline comment)

5. PR description layer counts are stale

The original description says "265 → 258" runtime / "~272 → 268" test_distroless. Subsequent commit bodies report more aggressive deltas: d43a50d says 258→245 and 268→255; 5f049be says 246→242 / ~252 on test_distroless. Post-squash, the runtime image is effectively a 1-layer filesystem (+ metadata), and the relevant number for the original test_distroless failure is now "layers downstream consumers inherit," not "layers in the runtime image" — different metric.

Refresh the description with the current architecture (one-layer squash for the runtime target) and the current numbers so reviewers and future readers don't have to read commit bodies in order.

6. jet lacks in-file context

The comment on lines 60-65 refers to "PyInstaller-bundled tools (jet)" without explaining what jet is. Commit 6e2b888c's body has the full story but commit bodies aren't searchable from the source tree. A one-line clarification — e.g., "jet is NVIDIA's internal PyInstaller-packaged CI runner" — would close the loop for an external reader.

7. Squash architecture — worth considering a split?

The PR started as three focused fixes (deps + CMD + workspace-collapse). It now also includes:

• The libstdc++ LD_PRELOAD workaround (6e2b888c)
• The FROM scratch + COPY / / squash + 50-var ENV reconstitution (a8087272 + 10188fbf)

The original three fixes are uncontroversial and ready to land. The squash introduces new mechanics (env reconstitution, runtime_full/runtime split) that have non-obvious risks (the inline comments cover several). Splitting the squash into its own PR would:

• Unblock the urgent fixes faster.
• Give the squash a focused review window with downstream consumers explicitly in the loop.
• Avoid bundling a behavioral change (squash) with bug fixes (deps/CMD).

Not a hard objection if downstream CI layer-cap pressure dictates landing it together — that's the author's call. But worth flagging.

8. Inline {% if %} in shell continuation — still ugly

Lines 103–105:

    && echo 'umask 002' > /etc/profile.d/00-umask.sh{% if target not in ("dev", "local-dev") %} \
    && python3 -m venv --system-site-packages /opt/dynamo/venv \
    && ln -sf /usr/bin/uv /opt/dynamo/venv/bin/uv{% endif %}

Jinja conditionals at the tail of a shell-continuation line are hard to scan. Considered low-priority but flagging again — the original separate {% if %} RUN ... {% endif %} block was easier to read.

---

Summary recommendation

Before merge:

1. Verify tileiras against actual upstream env (inline feat: nixl metadata store and retrieved from etcd #6).
2. Fix or test the aarch64 LD_LIBRARY_PATH (inline CI: Cache image on main branch #7).
3. Add libstdc++ existence assert (inline ci: update trigger token #2).
4. Confirm dev/local-dev builds + test_kvbm_imports.py runtime context (general ci: update container registry location #3, ci: remove timeout for cargo deny check #4).
5. Refresh PR description with current architecture and layer counts (general init: helm charts #5).
6. Rewrite or fold 10188fbf's commit message (general ci: update trigger token #2).

Also worth a deliberate decision on splitting the squash into a separate PR (general #7). Original three fixes look ready; the squash needs more scrutiny than it's gotten so far.

[tanmayv25]

@dmitry-tokarev-nv Reverting the squash. We believe a change on dynamo-ci should bring the layer counts just under threshold.

[dmitry-tokarev-nv]

Follow-up inline notes after the squash revert — concerns are tracked in the parent comment thread with bucket labels.

[dmitry-tokarev-nv]

🔴 Blocking — verify or fix before merge

1. Wheel persistence for dev/local-dev — possible regression (see inline comment on line 105)

• Pre-PR the wheel COPY was unconditional; now the cp is inside the non-dev Jinja guard.
• test_kvbm_imports.py greps the wheelhouse — needs confirmation it runs runtime-only.

2. Dev/local-dev/arm64 build verification (not line-anchored — belongs in test plan)

• Just three checkboxes added and exercised.

🟡 Recommended — fix in this PR

3. libstdc++ symlink target not asserted (see inline comment on line 79, with suggestion)

4. PR description is stale (not line-anchored — PR body refresh)

5. jet lacks in-file context (see inline comment on lines 57–58, with suggestion)

🟢 Nits — take or leave

6. librdmacm1 listed alongside rdma-core (prior inline comment on line 64 still applies — not re-posted)

7. chmod -R g+w /workspace is a no-op (see inline comment on line 146, with suggestion)

8. Inline {% if %} in shell-continuation (lines 96–98) (no inline — too stylistic for a clean suggestion)

[dmitry-tokarev-nv]

Major concerns addressed! Non-critical items below:

  Of my prior eight buckets, four are now closed (inline suggestions all accepted). Status of the remaining four:

  ┌─────┬──────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │  #  │             Item             │                                                  Status                                                  │
  ├─────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 2   │ Dev/local-dev/arm64 build    │ Still missing from test plan. Worth one explicit checklist round before merge — the dev path is now      │
  │     │ verification                 │ meaningfully different from the runtime path (dev keeps layered, runtime gets rebased).                  │
  ├─────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 4   │ PR description stale         │ Now even more stale — needs an "Architecture" paragraph mentioning the runtime_full + rebase split.      │
  ├─────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 6   │ librdmacm1 redundant with    │ Pedantic, ignored as expected.                                                                           │
  │     │ rdma-core                    │                                                                                                          │
  ├─────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 8   │ Inline {% if %} in shell     │ Still present (lines 104–106). Stylistic.                                                                │
  │     │ continuation                 │                                                                                                          │
  └─────┴──────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘