fix(examples): bump Triton example image to a CUDA-13 tag

[glamr-agent]

Overview:

examples/backends/tritonserver/Dockerfile copied a CUDA 12 Triton tree into a CUDA 13 Dynamo base, producing an ABI mismatch that docker build does not catch. This bumps the pinned Triton image from 25.01-py3 to 25.11-py3 so the copied tree matches the base's CUDA major version.

Details:

The Dockerfile is two-stage. FROM ${TRITON_SERVER_IMAGE} AS triton_source (line 7) supplies the tree that line 11 copies wholesale into ${DYNAMO_BASE_IMAGE}:

COPY --from=triton_source /opt/tritonserver /opt/tritonserver

container/context.yaml pins the Dynamo base at nvcr.io/nvidia/cuda-dl-base:25.11-cuda13.0-devel-ubuntu24.04 — CUDA 13, with no CUDA 12 variant. The old Triton pin 25.01-py3 is a CUDA 12 build, so tritonserver, libtritonserver.so, and the backend .so files all carried a libcudart.so.12 dependency that does not exist in the base. The copy succeeds and the failure surfaces at load time.

The change is one line:

-ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.01-py3"
+ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.11-py3"

Why 25.11-py3 specifically. Triton crosses to CUDA 13 at 25.08, so 25.08-py3 and later all satisfy the CUDA major requirement. 25.11 was chosen because three constraints line up on it:

• CUDA minor. 25.11-py3 is CUDA 13.0, matching the base's cuda13.0 exactly. 25.12 and later move to CUDA 13.1, which would put the copied tree a CUDA minor ahead of the base toolkit.
• NGC monthly train. Base and Triton both land on 25.11, so they share a driver floor, Ubuntu 24.04, and DCGM generation. This matters because lines 12-13 also copy /usr/local/dcgm and libdcgm*.so* across the stage boundary.
• CPython minor. Line 26 runs uv pip install /opt/tritonserver/python/triton*.whl — a compiled extension built for the source image's Python. r25.11 is Python 3.12, matching python_version: "3.12" in container/context.yaml.

25.08-py3 is the documented fallback if 25.11-py3 proves unsuitable; it is a one-string swap on the same line and requires no other change.

examples/backends/tritonserver/Dockerfile:5 is the only place in the repository that names this tag — grep -rIn 'TRITON_SERVER_IMAGE' returns only lines 5 and 7, and no README, compatibility matrix, Helm value, or lockfile restates it. The README documents no Triton tag, so no doc edit accompanies this.

Two related gaps found while investigating, deliberately left out of scope:

1. examples/backends/tritonserver/Makefile:29 clones triton-inference-server/server with no --branch or tag, so the from-source make all path tracks upstream main — a second, unpinned owner of the effective Triton version. Fixing it requires deciding a source-build policy.
2. No CI job builds this image. .github/workflows/ has no Triton reference and .github/filters.yaml has path filters for examples/backends/{vllm,sglang,trtllm,sample} but no tritonserver entry, which is why this drift went unnoticed.

Both are worth separate issues.

Where should the reviewer start?

examples/backends/tritonserver/Dockerfile:5 — that is the entire diff (1 file, 1 insertion, 1 deletion).

The check that closes this out is a build and a soname inspection, which requires a Docker daemon:

docker pull nvcr.io/nvidia/tritonserver:25.11-py3
cd examples/backends/tritonserver && docker build -t dynamo-triton:latest .
docker run --rm dynamo-triton:latest ldd /opt/tritonserver/bin/tritonserver | grep -i cudart

Expect libcudart.so.13, resolved, with no not found lines. That build was not run here — no Docker daemon was available in the environment this change was prepared in, and as noted above no CI job builds this image either. The CUDA 13 claim rests on upstream Triton release metadata and on build.py at the r25.01 and r25.11 branches, where the copied sonames move libcudart.so.12 → libcudart.so.13 in lockstep with libcublas, libcupti, and libnvJitLink.

One caution for anyone re-deriving the tag choice: cuda-dl-base and tritonserver are independent NGC families with different CUDA boundaries. cuda-dl-base crosses to CUDA 13 at 25.10; Triton crosses at 25.08. Reasoning from the base image's boundary would wrongly reject 25.08-py3.

pre-commit run --files examples/backends/tritonserver/Dockerfile --hook-stage manual passes (codespell, case conflicts, merge conflicts, shebang-executable, mixed line ending, trailing whitespace). The SPDX header is intact. A green linter does not prove runtime compatibility and is not offered as though it did.

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

_{Tracked internally as DYN-3697; no public GitHub issue exists for this.}

---

Summary by CodeRabbit

• Chores
  • Updated the Triton Server example environment to use the newer 25.11 release.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[glamr-agent]

Automated evidence record — validation incomplete

Evidence summary: [2/3 validated · 1 sandbox denied]

AI review assessment: needs_changes — advisory only. This is an automated AI agent's
judgment of whether the change is logically sound on the available evidence. It is not a
merge authorization; CI and a human reviewer hold that. The reviewer judged the one-line
diff technically correct, correctly scoped, and the right fix for the reported defect, and
recorded needs_changes solely because the validation did not pass.

Validation result: incomplete. The recorded run evidence shows validation did not pass.
The concrete reason is that the one check which would settle the claim — building the image
and inspecting the libcudart soname on the copied tritonserver binary — could not be run:
there is no docker binary on PATH in the preparation environment, DOCKER_HOST is unset,
and /var/run/docker.sock is absent. Deferring the proof to CI is not available either: no
workflow under .github/workflows/ references Triton and .github/filters.yaml has no
examples/backends/tritonserver entry, so no automated system produces this evidence
downstream. Separately, api.ngc.nvidia.com is off the environment's egress allowlist
(CONNECT tunnel failed, response 403), so publication of the nvcr.io/nvidia/tritonserver:25.11-py3
tag is likewise unconfirmed from the registry. The CUDA-13 claim rests on upstream Triton
release metadata and build.py at the r25.01 and r25.11 branches, reached via GitHub.

Evidence audit result: complete [2/3 validated · 1 sandbox denied] — the evidence table
is grounded in recorded runs. Every recipe named in the plan appears; none is missing. The
denied recipe is recorded as sandbox denied rather than dressed up as validated or waived
as N/A.

Evidence [2/3 validated · 1 sandbox denied]

Generated from validation/registry.jsonl — do not edit by hand.

Recipe	Status	Command	Evidence	Note
01-python-lint	validated	pre-commit run --files examples/backends/tritonserver/Dockerfile --hook-stage manual	validation/logs/2026-08-03T14-46-18.182Z-pre-commit-4e71.log
05-code-inspection	validated	bash -c ' set +e echo "=== Q1: r25.11 build.py context around libcudart COPY (lines 1560-1600) ==="; gh api repos/triton-inference-server/server/contents/build.py?ref=r25.11 --jq ".content" | base64 -d \ | sed -n "1560,1600p" | cat -n | awk "{printf \"%d\t%s\n\", \$1+1559, substr(\$0, index(\$0,\$2))}"; echo "Q1 exit=$?"; echo; echo "=== Q2: r25.01 build.py — ALL libcudart references ==="; gh api repos/triton-inference-server/server/contents/build.py?ref=r25.01 --jq ".content" | base64 -d \ | grep -nE "libcudart|libcublas|cuda/lib64"; echo "Q2 exit=$?"; echo; echo "=== Q3: r25.11 build.py — ALL libcudart references ==="; gh api repos/triton-inference-server/server/contents/build.py?ref=r25.11 --jq ".content" | base64 -d \ | grep -nE "libcudart|libcublas|cuda/lib64"; echo "Q3 exit=$?" '	validation/logs/2026-08-03T14-49-42.881Z-bash-6b6e.log
06-dockerfile-build	sandbox denied	docker info	—	docker info exits 127: no docker binary on PATH, DOCKER_HOST unset, /var/run/docker.sock absent. Recipe 06 preflight requires a reachable local Docker socket and has no remote-build fallback. N/A is not available here because CI does not own the proof: grep -rniE triton over 45 files in .github/workflows/ exits 1, and .github/filters.yaml has no examples/backends/tritonserver entry. The claim under validation is a container base-image ABI claim, so an image build is required by the claim.

[glamr-agent]

plan.md

# Plan — Triton example image is CUDA-12 while the Dynamo base is CUDA-13

Route: implementation
Template: dependency-upgrade
Engine: vllm

Base branch `main`; working branch `fix/triton-cuda13-abi-mismatch--1fdf61042fb3`.
Target: <https://linear.app/nvidia/issue/DYN-3697>. No review request was supplied in
`input.md`, and discovery found no existing request that implements this change, so no
`Review request:` line is recorded and no `Disposition:` line applies.

## User intent

`examples/backends/tritonserver/Dockerfile` builds a two-stage image. Line 5 pins
`ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.01-py3"`, and line 11 copies
`/opt/tritonserver` wholesale out of that stage into the Dynamo base selected by
`ARG DYNAMO_BASE_IMAGE`. The Dynamo base that `container/render.py` produces is pinned in
`container/context.yaml:13-15` at `nvcr.io/nvidia/cuda-dl-base:25.11-cuda13.0-devel-ubuntu24.04`
— CUDA 13. Triton `25.01-py3` is a CUDA-12 build, so every binary and shared object copied
across the stage boundary carries a `libcudart.so.12` (and sibling CUDA-12 soname)
dependency that does not exist in a CUDA-13 base. The result is a link failure at runtime,
not at build time: `docker build` happily copies the tree, and the breakage shows up when
`tritonserver`, `libtritonserver.so`, or a backend `.so` is first loaded.

The caller wants the pin bumped to a Triton release that is genuinely built against
CUDA 13, so that the copied tree resolves its sonames inside the Dynamo base. The caller
proposed `25.08-py3` "or newer".

## Non-goals

- **Not** changing `container/context.yaml` or the Dynamo base CUDA pin. The base is
  correct; the example's Triton pin is the stale side. Moving the base backwards to CUDA 12
  to satisfy this example would regress every framework image.
- **Not** pinning or otherwise repairing the from-source local-development path in
  `examples/backends/tritonserver/Makefile`. That Makefile clones
  `https://github.com/triton-inference-server/server.git` at line 29 with **no ref or tag**
  and builds with `build.py`, so it tracks upstream `main` and is a *second, unpinned*
  owner of the effective Triton version for the `make all` path. It is genuinely a latent
  reproducibility problem and a reviewer may well raise it, but it is a different defect
  from the container ABI mismatch, it is not what DYN-3697 asks for, and pinning it
  correctly requires deciding a source-build policy. Named here so it is not mistaken for
  an oversight.
- **Not** adding CI coverage for this image. No workflow builds it today (see Discovery);
  adding one is a separate, larger change.
- **Not** touching `examples/backends/tritonserver/src/`, `model_repo/`, or `launch/`. The
  ABI mismatch is entirely in the image composition.

## Discovery

Everything below is what I read, ran, or fetched — not what I intend to.

### The example tree, read in full

- `examples/backends/tritonserver/Dockerfile` (27 lines). Line 5 is the pin. Line 7
  `FROM ${TRITON_SERVER_IMAGE} AS triton_source`; line 9 `FROM ${DYNAMO_BASE_IMAGE} AS
  dynamo_base`; line 11 `COPY --from=triton_source /opt/tritonserver /opt/tritonserver`.
  Two consequences matter for the fix. First, lines 12-13 also copy `/usr/local/dcgm` and
  `/lib/x86_64-linux-gnu/libdcgm*.so*` from the same source stage, so DCGM comes along for
  the ride and is subject to the same glibc/CUDA expectations. Second — and easy to miss —
  **line 26** is `RUN uv pip install /opt/tritonserver/python/triton*.whl`, which installs
  the Triton Python bindings **from the copied tree**. That wheel contains a compiled
  extension linked against the source image's CUDA and built for the source image's CPython
  minor version. So the tag bump has to satisfy two ABI axes, not one: CUDA major, and
  CPython minor against the base's `python_version: "3.12"` (`container/context.yaml:36`).
- `examples/backends/tritonserver/README.md` (230 lines). Documents both the container path
  (`docker build -t dynamo-triton:latest .`, lines 40-48) and the `make all` source path
  (lines 71-91, 185-209). **It states no Triton image tag anywhere.** So the bump needs no
  README edit — a claim I checked by reading the file rather than assuming it.
- `examples/backends/tritonserver/Makefile` (105 lines). The unpinned clone at line 29,
  described under Non-goals. Contains no `25.01` string.
- `examples/backends/tritonserver/launch/identity.sh`. No image tag. Exports
  `LD_LIBRARY_PATH="${TRITON_DIR}/lib:${BACKEND_DIR}:..."` and launches
  `python3 -m dynamo.frontend --kserve-grpc-server ...` then `tritonworker.py`.
- `container/context.yaml` (150 lines). The base pin at lines 12-15. Also read the vllm,
  sglang, and trtllm blocks, because they are where the repo's other CUDA-13 evidence lives.

### Where the version is owned — repo-wide

A repo-wide, binary-excluded grep for the literal `25.01` over yaml/yml/md/mdx/Dockerfile/
sh/py/toml returned **exactly one** hit outside `.git/`:
`examples/backends/tritonserver/Dockerfile:5`. A grep for `TRITON_SERVER_IMAGE` over the
whole tree returned only Dockerfile lines 5 and 7. So this bump has **one** version owner.
There is no compatibility matrix, no lockfile, no Helm value, and no generated file that
restates the Triton tag. (Two earlier greps had to be re-run with `-I` and `--include`
filters because `docs/fern/assets/img/dynamo-logo.svg` path data matched the numeric
patterns and dumped a coordinate blob.)

Other `tritonserver` occurrences, checked and dismissed: `CODEOWNERS:237` and
`.github/codeowners/areas.yaml:231` (review routing), `src/tritonworker.py` (Python
import and endpoint names, no tag), and
`docs/fern/pages/reference/general/releases/dynamo-v1-2-0.mdx:320` (a historical release
note about PR #8697).

### CI ownership — the fact that decides Recipe 06's disposition

`grep -rniE "triton" .github/workflows/` returns **nothing**. `.github/filters.yaml` has
path filters for `examples/backends/{vllm,sglang,trtllm,sample}` but **no**
`examples/backends/tritonserver` entry, and `.github/labeler.yml` likewise. **No CI job
builds this image.** That is decisive below: I cannot honestly mark Recipe 06 `N/A` on the
grounds that "CI owns the image proof", because CI does not.

### Establishing the tag → CUDA mapping from evidence

The caller's `25.08-py3` claim needed independent verification, and repo evidence alone is
not merely insufficient here — it is **actively misleading**. The historical
`nvcr.io/nvidia/cuda-dl-base` tags that appear in `container/context.yaml` across git
history map `25.01→cuda12.8`, `25.06→cuda12.9`, `25.10→cuda13.0`, `25.11→cuda13.0`,
`25.12→cuda13.1`, `26.02→cuda13.1`. That family's CUDA-13 boundary is **25.10**. Anyone
reasoning by analogy from the base image family would conclude that `25.08` is CUDA 12 and
reject the caller's request. **`cuda-dl-base` and `tritonserver` are different NGC
families with independent CUDA boundaries; do not use one to reason about the other.**

Two attempts to consult the vendor-authoritative sources **failed**, and are recorded
verbatim rather than inferred away:

```
$ curl -sS -o /dev/null -w '%{http_code}\n' https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html
curl: (56) CONNECT tunnel failed, response 403
000

$ curl -sS -o /dev/null -w '%{http_code}\n' "https://api.ngc.nvidia.com/v2/repos/nvidia/tritonserver"
curl: (56) CONNECT tunnel failed, response 403
000
```

Both are allowlist denials — `docs.nvidia.com` and `api.ngc.nvidia.com` are not on the
sandbox egress allowlist. GitHub *is*, so I pivoted to upstream Triton's own release
metadata via `gh api`, which is authoritative for what went into each `YY.MM-py3` image
because the same repo produces it.

Upstream Triton's convention is `release vX.Y.Z` ↔ branch `rYY.MM` ↔ NGC tag `YY.MM-py3`.
I confirmed that binding directly rather than assuming it:

- `gh api repos/triton-inference-server/server/contents/build.py?ref=r25.08` →
  `DEFAULT_TRITON_VERSION_MAP = {"release_version": "2.60.0", "triton_container_version":
  "25.08", "upstream_container_version": "25.08", ..., "rhel_py_version": "3.12.3"}`.
- The same file at `ref=r25.11` → `"release_version": "2.63.0",
  "triton_container_version": "25.11"`, `"rhel_py_version": "3.12.3"`.
- `gh api repos/triton-inference-server/server/contents/Dockerfile.sdk?ref=r25.08` line 32:
  `ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.08-py3-min`; at `ref=r25.11`, the same
  line reads `...:25.11-py3-min`.

With the branch↔tag binding established, the release notes answer the CUDA question:

- **v2.60.0** (the r25.08 release, published 2025-08-26). Its "New Features and
  Improvements" section is one line: **"Added CUDA 13 support."** Its Known Issues discuss
  CUDA-13 fallout in the same release ("CuPy does not support CUDA 13 at the time of
  writing", linking `server/tree/r25.08`). Its ARM/SBSA section states "This release
  supports **CUDA** `13.0`, **TensorRT** `10.13.2.6`, ... **Python** `3.12`" and "This
  package is a subset of `nvcr.io/nvidia/tritonserver:25.08-py3`".
- v2.59.1 (r25.07) carries no CUDA-13 statement — so 25.08 is the boundary, not earlier.
- The ladder continues consistently: v2.63.0 → `25.11-py3`, CUDA `13.0`, Python `3.12`;
  v2.64.0 → `25.12-py3`, CUDA `13.1`; v2.65.0 → `26.01-py3`, CUDA `13.1`; v2.66.0 →
  `26.02-py3`, CUDA `13.1`, Python `3.12`.

Caveat on reading those notes: each release body contains **two** CUDA statements. The
Jetson/**iGPU** section of v2.60.0 says CUDA `12.9`; the SBSA/ARM section says CUDA `13.0`.
Only the latter is annotated as a subset of the `-py3` container. The iGPU number describes
a different artifact and must not be read as the `-py3` image's CUDA version.

**Conclusion: the caller's claim is verified — `25.08-py3` is the first CUDA-13 Triton
container — but the evidence is upstream GitHub release metadata, not repo evidence and
not NGC.** Residual uncertainty is stated honestly under "Confidence and what the printer
should do" below.

### Duplicate-work survey

`git log --since=2025-01-01 -- examples/backends/tritonserver/` (15 commits) and
`git log --since=2025-06-01 -- container/context.yaml` (15 commits). The example's own
history is: #4971 (created it, introducing the `25.01-py3` pin), #5794 / DYN-1984 (dynamo
base), #7037 + cherry-pick #7054 / DYN-2335 ("correct docker tag for tritonserver build"),
#8697 + cherry-pick #8871 / DYN-2883 (backend dir), then docs-only moves (#6700, #10855,
#12373) and unrelated sweeps (#6386, #6167, #6268, #9815, #10487).

#7037 is the one that looks like a duplicate by title, so I inspected it:
`git show e14be96a1` changes **only** `DYNAMO_BASE_IMAGE`. It left
`TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.01-py3"` untouched. Verified against
current `main` by reading Dockerfile line 5 as it stands today — still `25.01-py3`. So the
"correct docker tag" PR corrected a *different* tag; this work is not a repeat of it.

All-state host PR searches on `ai-dynamo/dynamo`, all of which **succeeded** (no auth or
network failure):

- `gh pr list --state all --search "DYN-3697"` → `[]`. No PR references the issue.
- `--search "TRITON_SERVER_IMAGE"` → only #4971 (MERGED), the PR that introduced the pin.
- `--search "tritonserver"` → 30 results; the only ones touching this example are
  #4971, #5794, #7037, #7054, #8697, #8871, all MERGED and all accounted for above. The
  remainder are unrelated (bindings, reasoning parsers, CODEOWNERS, operator CRDs).
- `--search "libcudart"` and `--search "triton cuda13"` → only unrelated sglang/vllm/
  container work; nothing touching this example.

**No open request competes with this change, and no merged request implements it.** Hence
`Route: implementation` (an ordinary implementation ask, not maintenance of a
factory-created request), `Template: dependency-upgrade` (not `existing-request`), and no
`Disposition: already-resolved`. Per `learnings/discovery-survey-before-planning.md`, this
survey was run before choosing the approach, not after.

## Chosen approach

Change one line. `examples/backends/tritonserver/Dockerfile:5` becomes:

```
ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.11-py3"
```

**Why `25.11-py3` rather than the caller's floor of `25.08-py3`.** Both satisfy the ask —
the caller said "25.08 or newer" and both are CUDA-13 by upstream attestation. `25.11-py3`
is the better choice on three grounds, each checked rather than assumed:

1. **CUDA minor alignment.** `25.11-py3` is CUDA `13.0` (v2.63.0 release notes). The Dynamo
   base is `25.11-cuda13.0-devel-ubuntu24.04` — also `13.0`. `25.08-py3` is also `13.0`, so
   both align; but `25.12-py3` and later are CUDA `13.1`, which would put the copied tree a
   CUDA minor *ahead* of the base's toolkit. `libcudart.so.13` keeps its soname across 13.x
   so it would very likely still load, but "very likely" is not a property to design a
   pinned example around. `25.11` is the newest tag that matches the base's CUDA minor
   exactly, and is therefore the right ceiling.
2. **Month alignment with the base.** Base and Triton both land on the `25.11` monthly
   train, so the two halves of the image come from the same NGC cadence — the same driver
   floor, the same Ubuntu 24.04, the same DCGM generation (`dcgm_version: 4.4.0-1` in
   r25.11's `build.py`), which matters because lines 12-13 copy DCGM across too.
3. **Python.** r25.11's `build.py` reports `rhel_py_version: 3.12.3` and v2.63.0's notes
   state Python `3.12`, matching `container/context.yaml`'s `python_version: "3.12"`. So
   the `triton*.whl` installed at Dockerfile line 26 is a cp312 wheel landing in a cp312
   interpreter. This axis is exactly as load-bearing as CUDA and is satisfied.

`25.08-py3` remains the documented **minimum** acceptable value and is the correct fallback
if `25.11-py3` turns out to be unavailable or otherwise unsuitable; the printer should say
so in `change.md` so a reviewer who prefers the caller's literal number knows why the plan
went one train further and that going back is safe.

Rollback point: the change is a single ARG string on one line. Reverting to
`25.01-py3` restores today's behavior exactly; there is no migration, no generated
artifact, and no other file to unwind.

The printer should also record in `change.md` — briefly, not as a second change — the
Makefile's unpinned clone (Non-goals) and the absence of any CI job building this image, so
the reviewer sees the two known gaps and does not mistake them for things the plan missed.

Commit with DCO (`git commit -s`) and a Conventional Commit subject; `fix(examples)` is the
right type and scope. The PR description needs `Summary` and `Validation` sections per
`AGENTS.md`, and the `Validation` section must state the Docker disposition plainly rather
than implying an image was built.

## Rejected alternatives

- **Bump to `25.08-py3` exactly, as proposed.** Correct and evidence-backed, and it stays
  literally inside the caller's words. Rejected as the primary choice only because it is
  three trains behind the base image for no gain; kept as the explicit documented minimum
  and fallback so nothing is lost if a reviewer prefers it. This is a preference the
  reviewer may reasonably overturn either way.
- **Bump to the newest available (`26.02-py3` or later).** Rejected: CUDA `13.1` against a
  `cuda13.0` base inverts the version skew, and the two newest upstream releases (v2.70.0 /
  26.06 and v2.71.0 / 26.07) **no longer state CUDA versions in their release notes at
  all**, deferring to the shipped container README — which I cannot read, since NGC is not
  reachable. Choosing a tag whose CUDA version I cannot attest would be exactly the
  confident guess this plan is supposed to avoid.
- **Move the Dynamo base back to CUDA 12 for this example.** Rejected: it would fork this
  example off the base every other image uses, and `container/context.yaml` has no CUDA-12
  `dynamo` variant left to point at.
- **Install Triton from `pip`/upstream wheels instead of copying `/opt/tritonserver`.**
  Rejected as out of scope: it redesigns the example's build (backends and the C++ server
  binary are not on PyPI), and it is not what DYN-3697 asks for.
- **Add a CUDA-12 compatibility shim (`cuda-compat`, or symlinking `libcudart.so.12`).**
  Rejected: soname aliasing across a CUDA major is not supported and would swap a loud,
  immediate load failure for a quiet, undefined one.
- **Also pin the Makefile's clone in the same change.** Rejected per Non-goals: a real but
  distinct defect, and bundling it would make a one-line fix into a policy decision.

## Validation strategy

Being honest about what this change is: a single `ARG` string in a Dockerfile, in a
sandbox with **no Docker socket** (`compute-env.md`: "_Docker daemon unavailable — recipe
06 (dockerfile-build) is not runnable in this sandbox._"). The one piece of evidence that
would settle the claim beyond argument — build the image and run `ldd` on
`/opt/tritonserver/bin/tritonserver` inside it — cannot be produced here. Nothing else can
substitute for it, and per `learnings/no-tautological-tests.md` I will not dress up
unrelated green checks as though they did.

So the ladder is deliberately small, and the one recipe that cannot run carries a planned,
recorded disposition rather than being quietly omitted.

**`01-python-lint`** — Section 1 is `N/A` (no Python files touched; files: the single
`examples/backends/tritonserver/Dockerfile`). Section 2 is the substance: `pre-commit run
--files examples/backends/tritonserver/Dockerfile --hook-stage manual`. The repo's hooks
that apply to this file are the `pre-commit-hooks` set — `check-merge-conflict`,
`mixed-line-ending`, `trailing-whitespace` — plus `codespell`. This proves the edit is
well-formed and does not disturb the SPDX header at lines 1-2. It is a pre-screen, not a
proof of runtime behavior; the packet must say so. No hardware needed.

**`05-code-inspection`** — this is where the actual claim is defended, and it is the right
recipe precisely because the claim is provable by reading. Section 1 is `N/A` (there is no
upstream PR number to diff; the diff is this work item's own `change.diff`). Section 2 is
the call-path trace, and each behavioral claim has a citation the validator can re-check
independently:
  - *Claim:* the CUDA-12 tree crosses into a CUDA-13 base. *Evidence:*
    `examples/backends/tritonserver/Dockerfile:7` and `:11` (`COPY --from=triton_source
    /opt/tritonserver /opt/tritonserver`) against `container/context.yaml:15`
    (`base_image_tag: 25.11-cuda13.0-devel-ubuntu24.04`).
  - *Claim:* the Python bindings are subject to the same mismatch. *Evidence:*
    `examples/backends/tritonserver/Dockerfile:26`.
  - *Claim:* the new tag is CUDA 13 and cp312. *Evidence:* upstream
    `build.py@r25.11` `DEFAULT_TRITON_VERSION_MAP` (`release_version: 2.63.0`,
    `triton_container_version: 25.11`, `rhel_py_version: 3.12.3`), release v2.63.0's
    "supports **CUDA** `13.0` ... **Python** `3.12`" line and its
    "subset of `nvcr.io/nvidia/tritonserver:25.11-py3`" line, and
    `Dockerfile.sdk@r25.11:32` binding branch `r25.11` to tag `25.11-py3`. The validator
    should re-fetch these through `gh api` rather than trusting this plan's transcription.
  Section 3 is the parallel-pattern check, and it has a concrete form here: re-run the
  repo-wide grep for `25.01` and `TRITON_SERVER_IMAGE` and confirm the changed line is the
  only owner — i.e. confirm this is not a half-fix with a stale tag left in a README or a
  script. Section 4 (`gh pr checks`) is `N/A` pre-publication. No hardware needed.

**`06-dockerfile-build`** — **nominated with a planned disposition of `blocked /
no-docker-socket`, recorded as `sandbox-denied`.** The reasoning, stated so the validator
does not have to re-derive it: `compute-env.md` offers `N/A` "when CI owns the
Dockerfile/image proof". Discovery shows CI does **not** — `.github/workflows/` contains no
job mentioning Triton, and `.github/filters.yaml` has no
`examples/backends/tritonserver` path filter. Marking it `N/A` would therefore be a false
claim that some other system will produce the evidence. The honest record is that the
proof is unavailable, so the validator runs `docker info >/dev/null`, records the failure
through the recorder as `sandbox-denied` per Recipe 06's own text, and the overall verdict
for this work item is `blocked / no-docker-socket`. Per
`learnings/blocked-validation-is-terminal.md` that verdict is terminal: the agency must not
loop on it, and the work publishes as `[blocked]` with the ABI reasoning and the upstream
citations standing on their own. The dependency-upgrade template's warning that "a green
linter alone does not prove runtime compatibility" is exactly the situation, and the packet
should say so in those terms rather than implying more coverage than exists.

Recipes deliberately **not** nominated, with reasons, so the omissions read as decisions:
`00-dynamo-editable-install` (nothing imports `dynamo.*`; the diff touches no Python);
`02-rust-cargo-check` (no Rust); `03-python-unit-tests-mocker` and `04-python-runtime-lint`
(no Python source or runtime behavior changes); `07-agg-smoke` and `08-disagg-pair-smoke`
(the sandbox engine is vLLM and **nothing on the vLLM path changes** — a green vLLM smoke
would be perfectly tautological here, and the Triton path cannot be smoked because the
image can neither be built nor pulled); `09-gpu-pytest` (no GPU-facing code changes);
`10-perf-benchmark` (no performance claim); `11-go-operator-tests` (no operator changes).
The A100 attached to this sandbox is not needed by any nominated recipe.

```validation-recipes
01-python-lint
05-code-inspection
06-dockerfile-build
```

### Confidence, and what the printer should do about the residual uncertainty

Confidence that `25.08-py3` and `25.11-py3` are CUDA-13 Triton containers: **high**,
sourced from upstream Triton's own release metadata on GitHub (v2.60.0's "Added CUDA 13
support.", v2.63.0's "supports **CUDA** `13.0`"), cross-checked against `build.py`'s
version map and `Dockerfile.sdk`'s base-image ARG on the matching branches. It is **not**
sourced from repo evidence — repo evidence points the wrong way, because `cuda-dl-base`
crosses to CUDA 13 at `25.10` — and it is **not** sourced from NGC, which returned 403.

Two things I could not verify and will not claim:

1. **The actual NGC image contents were never inspected.** `api.ngc.nvidia.com` is off the
   allowlist and there is no Docker socket, so I could not confirm that
   `nvcr.io/nvidia/tritonserver:25.11-py3` exists as a published tag, nor read its
   `libcudart` soname. Upstream release notes describe what the release *is*; only the
   registry proves what was *published*. The printer must not write anything in `change.md`
   implying the image was pulled, inspected, or built.
2. **Tag availability.** If `25.11-py3` is for any reason not published, the fallback is
   `25.08-py3` — the caller's own floor, and the first CUDA-13 release. The printer should
   note this fallback explicitly in `change.md` so a reviewer with registry access can make
   the swap in one line without re-deriving the reasoning.

The PR body's `Validation` section must state, in plain words, that the image was not built
locally because the sandbox has no Docker daemon, that no CI job builds this image either,
and that the CUDA-13 claim rests on upstream Triton release metadata — so the first human
reviewer knows exactly which single check they need to run to close the loop.

## Required deliverables

- `change.md` — the narrative: the ABI mismatch, the single version owner, why `25.11-py3`
  (with `25.08-py3` recorded as the documented minimum and fallback), the rollback point,
  and the two out-of-scope observations (unpinned Makefile clone; no CI builds this image).
- `change.diff` — the one-line edit to `examples/backends/tritonserver/Dockerfile:5`, on
  branch `fix/triton-cuda13-abi-mismatch--1fdf61042fb3` off `main`, DCO-signed.
- `change-validation.md` — recorded runs for `01-python-lint` and `05-code-inspection`, and
  the recorded `sandbox-denied` disposition for `06-dockerfile-build`, with the overall
  verdict `blocked / no-docker-socket`.
- `review.md` — independent review of the diff and the evidence, per
  `learnings/reviewer-independent-of-validator.md`.
- A published pull request against `ai-dynamo/dynamo`, Conventional Commit title
  (`fix(examples): ...`), body carrying `Summary` and `Validation`, referencing DYN-3697,
  and published as `[blocked]` per `learnings/publish-every-terminal-verdict.md`.

[glamr-agent]

change.md

# Change — Triton example image bumped to a CUDA-13 tag

Branch `fix/triton-cuda13-abi-mismatch--1fdf61042fb3`, branched from `main` at
`70544474680e7924aa7f0ae95b1e72a83bf40bf8`.

Commit: **`8f41c2cbeeeb992f04e1bd7282e73221abc17b59`** — `fix(examples): bump Triton example
image to a CUDA-13 tag`. DCO-signed; author `svc-glamr@nvidia.com` and the
`Signed-off-by:` trailer match (checked with
`git log -1 --format='%ae | %(trailers:key=Signed-off-by,valueonly)'`).

`change.diff` was captured with `git diff main...HEAD` and verified to apply cleanly at the
branch point by `git apply --check` inside a throwaway worktree checked out at `main`.

## What changed

One file, one line.

`examples/backends/tritonserver/Dockerfile:5`

```
-ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.01-py3"
+ARG TRITON_SERVER_IMAGE="nvcr.io/nvidia/tritonserver:25.11-py3"
```

That is the entire diff: 1 file changed, 1 insertion, 1 deletion. Nothing else in the tree
was touched.

## Why this fixes the reported failure

The Dockerfile builds two stages. Line 7 is `FROM ${TRITON_SERVER_IMAGE} AS triton_source`;
line 9 is `FROM ${DYNAMO_BASE_IMAGE} AS dynamo_base`; line 11 is
`COPY --from=triton_source /opt/tritonserver /opt/tritonserver`. The Dynamo base is pinned
in `container/context.yaml:15` at `base_image_tag: 25.11-cuda13.0-devel-ubuntu24.04` under
`base_image: nvcr.io/nvidia/cuda-dl-base` — CUDA 13. The old Triton pin, `25.01-py3`, is a
CUDA-12 build, so every binary and shared object copied across the stage boundary carried a
`libcudart.so.12` (and sibling CUDA-12 soname) dependency that does not exist in a CUDA-13
base. The mismatch is invisible to `docker build`, which happily copies the tree; it
surfaces at load time, when `tritonserver`, `libtritonserver.so`, or a backend `.so` is
first opened.

Two details in the same file constrain which replacement tag is acceptable:

- Lines 12-13 also copy `/usr/local/dcgm` and `/lib/x86_64-linux-gnu/libdcgm*.so*` from the
  same source stage, so DCGM crosses the boundary too and inherits the same expectations.
- Line 26 is `RUN uv pip install /opt/tritonserver/python/triton*.whl` — the Triton Python
  bindings are installed **from the copied tree**. That wheel carries a compiled extension
  built for the source image's CPython minor version. So the bump has to satisfy two ABI
  axes, CUDA major *and* CPython minor, not one.

## Single version owner — verified, not assumed

I re-ran the plan's greps myself rather than trusting them:

- `grep -rInE '25\.01'` restricted to yaml/yml/md/mdx/Dockerfile\*/sh/py/toml, excluding
  `.git/`, returns **exactly one** hit: `examples/backends/tritonserver/Dockerfile:5`.
- `grep -rIn 'TRITON_SERVER_IMAGE'` over the tree returns only Dockerfile lines 5 and 7.
- `grep -nE '25\.[0-9]{2}|tritonserver:' examples/backends/tritonserver/README.md` returns
  nothing — the README states no Triton tag, so no README edit is needed. (The plan asserted
  this; I confirmed it independently before accepting it.)

There is no lockfile, compatibility matrix, Helm value, or generated file restating the tag.
One owner, one line.

## Why `25.11-py3` rather than the caller's floor of `25.08-py3`

The caller asked for "25.08-py3 or newer". Both satisfy the ask. `25.11-py3` was chosen on
three grounds:

1. **CUDA minor alignment.** `25.11-py3` is CUDA `13.0`; the Dynamo base is
   `25.11-cuda13.0-devel-ubuntu24.04`, also `13.0`. `25.08-py3` is `13.0` too, so both
   align — but `25.12-py3` and later move to CUDA `13.1`, which would put the copied tree a
   CUDA minor *ahead* of the base's toolkit. `25.11` is the newest tag matching the base's
   CUDA minor exactly, so it is the right ceiling.
2. **Same NGC monthly train as the base.** Base and Triton both land on `25.11`, so the two
   halves of the image share a driver floor, Ubuntu 24.04, and DCGM generation
   (`dcgm_version: 4.4.0-1` in r25.11's `build.py`). This matters specifically because
   Dockerfile lines 12-13 copy DCGM across.
3. **CPython minor.** r25.11's `build.py` reports `rhel_py_version: "3.12.3"` and release
   v2.63.0's notes state Python `3.12`, matching `container/context.yaml:36`
   `python_version: "3.12"`. So the `triton*.whl` installed at line 26 is a cp312 wheel
   landing in a cp312 interpreter — the second ABI axis is satisfied.

**`25.08-py3` is recorded here as the documented minimum acceptable value and the fallback.**
It is the caller's own floor and the first CUDA-13 Triton container. If `25.11-py3` turns
out to be unpublished or otherwise unsuitable, a reviewer with registry access can swap it
in on this one line without re-deriving any of the reasoning above; nothing else in the diff
depends on which of the two is chosen.

## Rollback point

Revert the single `ARG` string on `examples/backends/tritonserver/Dockerfile:5` back to
`nvcr.io/nvidia/tritonserver:25.01-py3`. That restores today's behavior exactly. There is no
migration step, no generated artifact, and no other file to unwind.

## Evidence for the CUDA-13 claim — and what was *not* done

I re-fetched every upstream citation myself through `gh api` rather than transcribing the
plan's:

- `repos/triton-inference-server/server/contents/build.py?ref=r25.11` →
  `DEFAULT_TRITON_VERSION_MAP` with `"release_version": "2.63.0"`,
  `"triton_container_version": "25.11"`, `"dcgm_version": "4.4.0-1"`, and (line 82)
  `"rhel_py_version": "3.12.3"`.
- `contents/Dockerfile.sdk?ref=r25.11` line 32 →
  `ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.11-py3-min`, which is what binds upstream
  branch `r25.11` to the NGC tag family `25.11-py3`.
- Release `v2.63.0` ("Release 2.63.0 corresponding to NGC container 25.11", published
  2025-11-26) states "This release supports **CUDA** `13.0` ... **Python** `3.12`" and, two
  lines later, "This package is a subset of `nvcr.io/nvidia/tritonserver:25.11-py3`".
- Release `v2.60.0` ("Release 2.60.0 corresponding to NGC container 25.08"): its **New
  Features and Improvements** section is the single line "Added CUDA 13 support." Its
  corresponding section states CUDA `13.0` and "subset of
  `nvcr.io/nvidia/tritonserver:25.08-py3`".
- Release `v2.59.1` (r25.07) contains no CUDA-13 statement at all — so `25.08` is the
  boundary, not something earlier.

**One correction to the plan's transcription, recorded rather than smoothed over.** The plan
attributes v2.63.0's CUDA `13.0` line to an "ARM/SBSA section". In the body I fetched, that
line sits under a `<summary><h2>Jetson AGX Systems Support</h2></summary>` block, not one
labelled SBSA. The load-bearing part of the citation is unaffected: the CUDA `13.0` line is
the one immediately followed by "subset of `nvcr.io/nvidia/tritonserver:25.11-py3`", which
is what ties it to the `-py3` container. The plan's substantive caveat also holds and is
worth repeating — each release body carries **two** CUDA statements, and the other one
(under `Jetson iGPU Support`) says CUDA `12.9`. That iGPU number describes a different
artifact and must not be read as the `-py3` image's CUDA version.

**A trap worth flagging for the reviewer.** `cuda-dl-base` and `tritonserver` are different
NGC families with independent CUDA boundaries. `cuda-dl-base` crosses to CUDA 13 at `25.10`
(visible in this repo's own history of `container/context.yaml`). That says nothing about
Triton, which crosses at `25.08`. Reasoning by analogy from the base-image family would
wrongly conclude `25.08-py3` is CUDA 12 and reject the caller's request.

### What I could not verify

- **The image was not built, pulled, or inspected.** `docker` is not on `PATH` in this
  sandbox at all (`command -v docker` → not found), so there is no `ldd` evidence on
  `/opt/tritonserver/bin/tritonserver` and no build proof. This is the one check that would
  settle the claim beyond argument, and it does not exist here.
- **NGC was not consulted.** I re-ran the reachability check myself:
  `curl -sS -o /dev/null -w '%{http_code}\n' "https://api.ngc.nvidia.com/v2/repos/nvidia/tritonserver"`
  → `curl: (56) CONNECT tunnel failed, response 403`, `000`. `api.ngc.nvidia.com` and
  `docs.nvidia.com` are off this sandbox's egress allowlist. So I have **not** confirmed that
  `nvcr.io/nvidia/tritonserver:25.11-py3` exists as a published tag, and I have not read its
  `libcudart` soname. Upstream release notes describe what a release *is*; only the registry
  proves what was *published*.

Honest framing of the whole evidence chain: the CUDA-13 claim rests on upstream Triton's own
release metadata reached via GitHub, cross-checked against `build.py` and `Dockerfile.sdk` on
the matching branches. It does **not** rest on repo evidence (which points the wrong way, per
the trap above) and it does **not** rest on NGC. The single check a human reviewer needs to
close the loop is: pull `nvcr.io/nvidia/tritonserver:25.11-py3`, build this Dockerfile
against the CUDA-13 base, and run `ldd /opt/tritonserver/bin/tritonserver`.

### I tried X, fell back to Y

- Tried to reach NGC's registry API to confirm tag publication and inspect sonames. Got
  `CONNECT tunnel failed, response 403` (allowlist denial). Fell back to upstream Triton's
  GitHub release metadata via `gh api`, which is authoritative for what went *into* each
  `YY.MM-py3` image because the same repo produces it — but is not proof of publication.
- Tried to confirm the Docker build path locally. There is no `docker` binary in this
  sandbox, so no fallback exists; the build proof is simply absent, and is not substituted
  for by any other green check.

## Lint

`pre-commit run --files examples/backends/tritonserver/Dockerfile --hook-stage manual` →
exit 0. Applicable hooks passed: `codespell`, `check for case conflicts`, `check for merge
conflicts`, `check that scripts with shebangs are executable`, `mixed line ending`, `trim
trailing whitespace`, `Report pytest markers`. Python/Rust/JSON/YAML hooks were skipped as
"no files to check" — correct, since the diff touches no Python, Rust, or structured-config
file. The SPDX header at lines 1-2 is intact.

Per the dependency-upgrade template's own warning: a green linter alone does not prove
runtime compatibility, and it is not offered here as though it did. No build, no test, and
no smoke run exercises this image in this sandbox.

## Out-of-scope observations — NOT part of this change

Both are recorded so the reviewer sees them as known gaps rather than oversights. **Neither
is touched by this diff.**

1. **`examples/backends/tritonserver/Makefile:29` clones upstream with no ref or tag.** The
   line is `cd /tmp && git clone https://github.com/triton-inference-server/server.git`,
   with no `--branch`/`--depth`/tag. The `make all` from-source path therefore tracks
   upstream `main` and is a *second, unpinned* owner of the effective Triton version. This is
   a genuine reproducibility problem, but it is a different defect from the container ABI
   mismatch, it is not what DYN-3697 asks for, and pinning it correctly requires deciding a
   source-build policy. Left alone deliberately.
2. **No CI job builds this image.** `grep -rniE 'triton' .github/workflows/` returns no match
   (exit 1). `.github/filters.yaml` has path filters for
   `examples/backends/{vllm,sglang,trtllm,sample}` but no `examples/backends/tritonserver`
   entry (grep exit 1), and `.github/labeler.yml` likewise. This matters for how the change is
   reviewed: no automated system will produce the build proof that this sandbox could not, so
   the Dockerfile-build evidence is genuinely unavailable rather than deferred to CI. Adding
   such a job is a separate, larger change.

MR URL: https://github.com/ai-dynamo/dynamo/pull/12569

[glamr-agent]

change-validation.md

# Validation — Triton example image bumped to a CUDA-13 tag

Branch `fix/triton-cuda13-abi-mismatch--1fdf61042fb3`, commit
`8f41c2cbeeeb992f04e1bd7282e73221abc17b59`. One file, one line:
`examples/backends/tritonserver/Dockerfile:5`, `nvcr.io/nvidia/tritonserver:25.01-py3` →
`25.11-py3`.

Three recipes were nominated and all three were run: `01-python-lint`,
`05-code-inspection`, `06-dockerfile-build`. Every command below went through the
recorder; the coverage table is generated from `validation/registry.jsonl` and
`validation/dispositions.jsonl`, not written by hand.

## Recipe 01 — Python lint

Section 1 is `N/A`: the diff contains no Python. Section 2 ran
`pre-commit run --files examples/backends/tritonserver/Dockerfile --hook-stage manual`
→ **exit 0**. `codespell`, case-conflict, merge-conflict, shebang-executable, mixed
line ending, trailing whitespace, and the pytest-marker report all passed; the
Python/Rust/JSON/YAML hooks reported "no files to check", which is correct for a
Dockerfile-only diff.

**This is a pre-screen, not proof of runtime behavior.** It establishes only that the
edited line is well-formed text. A linter cannot observe a CUDA soname, and nothing in
this recipe touches the claim under validation.

## Recipe 05 — Code inspection

This is where the claim is actually defended, and it is the only recipe that carries
real weight here. Section 1 is `N/A` (no upstream PR under review). Section 4 is `N/A`
(pre-publication). Sections 2 and 3 were run in full.

I re-derived each cited claim myself rather than transcribing the plan's or the
printer's version. Two of the three re-checks changed the picture.

### Section 2 — call-path trace

**(a) The CUDA-12 tree does cross into a CUDA-13 base.** Reading the Dockerfile in
full: line 7 `FROM ${TRITON_SERVER_IMAGE} AS triton_source`, line 9
`FROM ${DYNAMO_BASE_IMAGE} AS dynamo_base`, then line 11
`COPY --from=triton_source /opt/tritonserver /opt/tritonserver`, plus lines 12–13
copying `/usr/local/dcgm` and `/lib/x86_64-linux-gnu/libdcgm*.so*`. The base side is
pinned in `container/context.yaml:15` at `base_image_tag: 25.11-cuda13.0-devel-ubuntu24.04`
under `base_image: nvcr.io/nvidia/cuda-dl-base`. The `dynamo` framework block in that
file has exactly **one** device key, `cuda13.0` — so a `dynamo` render is
unconditionally CUDA 13, with no CUDA-12 variant reachable. `render.py`'s `--device`
default is `"cuda"` and it reads `context.yaml` as its source of truth.
`examples/backends/tritonserver/README.md:43` binds `DYNAMO_BASE_IMAGE` to that
rendered output. The two halves of the copy are therefore genuinely on opposite sides
of a CUDA major boundary. Confirmed.

**(b) The Python bindings share the mismatch.** Line 26 is
`RUN uv pip install /opt/tritonserver/python/triton*.whl` — the wheel is installed
*from the copied tree*, so it carries the source image's compiled extension. That makes
CPython minor a second ABI axis alongside CUDA major. Confirmed.

**(c) The new tag is CUDA 13 and cp312.** Re-fetched through `gh api` at the pinned
refs. `build.py@r25.11` `DEFAULT_TRITON_VERSION_MAP` gives `"release_version": "2.63.0"`,
`"triton_container_version": "25.11"`, `"rhel_py_version": "3.12.3"`.
`Dockerfile.sdk@r25.11:32` is `ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.11-py3-min`,
which is what binds upstream branch `r25.11` to the NGC tag family `25.11-py3`; the
same line at `r25.01` reads `25.01-py3-min`. Release names confirm the convention
(v2.63.0 "corresponding to NGC container 25.11", v2.54.0 "…25.01"). The 25.08 boundary
is independently visible: v2.60.0's entire "New Features and Improvements" section is
the single line "Added CUDA 13 support.", and v2.59.1 (r25.07) contains no CUDA
statement at all (grep exit 1).

**The printer's correction is confirmed; the plan's transcription is refuted.** The plan
attributed v2.63.0's CUDA `13.0` line to an "ARM/SBSA section". In the body I fetched,
that line (body line 141) sits under `<summary><h2>Jetson AGX Systems Support</h2>`
(line 137). The printer caught this and recorded it; the plan is wrong on this point.

**A further nuance neither the plan nor the printer stated.** Body line 143 reads "This
package is a subset of `nvcr.io/nvidia/tritonserver:25.11-py3` **ARM container image**".
The "subset of" sentence — the exact hinge both documents leaned on to tie the CUDA
`13.0` number to the `-py3` container — is scoped to the **ARM** image. This Dockerfile
copies `/lib/x86_64-linux-gnu/...`, i.e. x86_64. v2.60.0 carries the same ARM scoping
(line 121, for `25.08-py3`). So that citation is weaker for the x86 case than either
document represented. I avoided the trap flagged in the same body: line 122's CUDA
`12.9` sits under `Jetson iGPU Support` (line 118) and describes the igpu tar, not the
`-py3` container.

**Stronger, x86-relevant substitutes were located, and they hold the claim up.**
`docs/introduction/compatibility.md@v2.63.0` gives per-tag CUDA versions:
`25.11-trtllm-python-py3` → CUDA `13.0.2.006`, Python 3.12.3; `25.11-vllm-python-py3` →
CUDA `13.0.2.006`. The old pin's row: `25.01-trtllm-python-py3` → CUDA `12.8.0.038`. The
boundary is visible in the same table (`25.07` → `12.9.0.043`, `25.08` → `13.0.1.012`).
Caveat: that file has no plain `-py3` section, only backend-suffixed variants.

The most direct evidence is in `build.py` itself. At `r25.01:1550`:
`COPY --from=min_container /usr/local/cuda/lib64/libcudart.so.12 ...`. At `r25.11:1582`
the same line reads `libcudart.so.13`. The full set moves together —
`libcupti.so.12→.13`, `libnvJitLink.so.12→.13`, `libcublas`/`libcublasLt.so.12→.13` —
and `patchelf --add-needed .../libcublasLt.so.12` (r25.01:1258) becomes `.so.13`
(r25.11:1269); r25.11:1528 installs `libnvshmem3-cuda-13`. This is soname-level
evidence for both halves of the claim: the old pin really did carry CUDA-12 sonames,
and the new one carries CUDA-13. Caveat: these lines live in
`add_cpu_libs_to_linux_dockerfile` and are arch-parameterized
(`cuda_arch = "sbsa" if aarch64 else "x86_64"`), so they describe both arches rather
than x86 exclusively.

### Section 3 — parallel patterns, single version owner

Re-ran the greps myself, **unpiped**, because piping to `head` masks grep's exit code —
my own first pass made exactly that error and reported the pipeline's status instead of
grep's. Corrected results: `25.01` no longer appears anywhere in the tree (excluding
`.git` and, per the plan's warning, the coordinate blob in
`docs/fern/assets/img/dynamo-logo.svg`) — **grep exit 1**. `TRITON_SERVER_IMAGE` appears
only at Dockerfile lines 5 and 7. The README states no Triton tag. This is a complete
fix, not a half-fix leaving a stale tag in a README or script.

## Recipe 06 — Dockerfile build

**The premise was verified, not assumed.** `compute-env.md` allows `N/A` "when CI owns
the Dockerfile/image proof", so that escape hatch had to be tested before it could be
declined. `grep -rniE "triton" .github/workflows/` → **exit 1**, across 45 workflow
files (counted in the same recorded run, so the directory is demonstrably non-empty).
`.github/filters.yaml` has no `examples/backends/tritonserver` entry → **exit 1**, while
filters do exist for the vllm/sglang/trtllm/sample backends. The only `.github/` hit
anywhere is `codeowners/areas.yaml:231`, which is review routing, not a build. **CI does
not own this proof.** No automated system downstream will produce the build evidence
this sandbox cannot.

Preflight per the recipe: `docker info` → **exit 127**. Not a permission error — the
binary is absent. `command -v docker` finds nothing, `DOCKER_HOST` is unset, and
`/var/run/docker.sock` does not exist. Recorded as **`sandbox-denied`**, which is what
Recipe 06's own text prescribes when the socket is not mounted. The recipe states plainly
that there is no remote-build fallback.

## What this validation does and does not establish

The change is internally coherent and correctly scoped: one owner, one line, no stale
references, a rollback that is a single string revert, and an argument that survives
independent re-derivation. The `libcudart.so.12`→`.so.13` transition in upstream's own
`build.py` substantiates the ABI-mismatch diagnosis and the direction of the fix.

What is missing is the one check that would settle it: **no image was built, pulled, or
inspected.** There is no `ldd /opt/tritonserver/bin/tritonserver` output. Separately,
NGC is off this sandbox's egress allowlist (`api.ngc.nvidia.com` →
`CONNECT tunnel failed, response 403`), so **it remains unconfirmed that
`nvcr.io/nvidia/tritonserver:25.11-py3` exists as a published tag.** Upstream release
metadata describes what a release *is*; only the registry proves what was *published*.
The printer stated this limitation plainly rather than papering over it, and I reached
the same conclusion independently.

The claim under validation is a container base-image ABI claim. An image build is
required by the claim, and it is unavailable. Per the recipe, that makes the overall
verdict `blocked` — and this is terminal, not a retry: no amount of re-running changes
whether a Docker socket exists. I did not substitute unrelated green checks (for
instance a vLLM smoke) to pad the packet; nothing on the vLLM path changed, and such a
run would prove nothing about this diff.

The single check a human reviewer needs to close the loop: pull
`nvcr.io/nvidia/tritonserver:25.11-py3`, build this Dockerfile against the CUDA-13 base,
and run `ldd` on the copied `tritonserver` binary.

## Verdict: blocked

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔍 Wheel glob and DCGM copy paths depend on the new image's layout

The image bump changes the contents of the source stage, and two later steps depend on that layout in ways the build cannot verify locally: COPY --from=triton_source /usr/local/dcgm ... and /lib/x86_64-linux-gnu/libdcgm*.so* (lines 12-13) fail the build if those paths were dropped or renamed in the newer Triton image, and uv pip install /opt/tritonserver/python/triton*.whl (line 26) glob-matches whatever wheels the new image ships (historically both tritonserver-*.whl and tritonfrontend-*.whl live there). Since the PR description states the build was not executed and no CI job builds this image, these are the concrete points to verify in a manual build before merging.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[glamr-agent]

review.md

> 🤖 **Automated AI review — advisory.** An AI agent's judgment of
> whether this change is logically sound on the evidence; not a merge
> authorization. CI, customs, and a human reviewer hold that.

## Assessment: needs_changes

**Read this heading carefully — it is not a statement that the diff is wrong.**

The one-line diff is, in my independent judgment, **technically correct, correctly
scoped, and the right fix for the reported defect**, and the packets are **honest about
what was and was not proven**. I re-derived the mechanism and the upstream tag→CUDA
mapping from primary sources myself and reached the same conclusion the plan and printer
did.

The assessment is `needs_changes` for one reason only: `change-validation.md` closes with
`## Verdict: blocked`, and the reviewer's deterministic checklist item
`validator-blocked-or-failed-implies-request-changes` (`blocker_allowed: false`) states
that "the reviewer never assesses a change `sound` over a blocked, failed, or hedged
validation." The empirical case — build the image, run `ldd` on the copied
`tritonserver` binary — was never made, because this sandbox has no Docker binary and no
CI job builds this image. I have no standing to invent that evidence on the validator's
behalf.

Per `learnings/blocked-validation-is-terminal.md` this assessment **records findings; it
does not drive another agency iteration**. Nothing a further iteration does can conjure a
Docker socket. The correct disposition is to publish as `[blocked]` / *validation
incomplete*, with the single closing check named for the human reviewer.

---

## Findings

| # | Location | Severity | Claim | Evidence |
|---|---|---|---|---|
| 1 | `change-validation.md:156` | blocking | Validation verdict is `blocked`; the ABI claim has no runtime proof. Bars a `sound` assessment under the reviewer checklist. | Verdict line is a literal, unhedged `blocked` — not "conditionally passes". Customs table reads `[2/3 validated · 1 sandbox denied]`; `06-dockerfile-build` is `sandbox denied`, not dressed up as validated. Its log (`validation/logs/2026-08-03T14-52-34.727Z-docker-bb19.log`) contains exactly `evidence record: spawn failed: spawn docker ENOENT` — a genuine denial, correctly recorded. |
| 2 | `examples/backends/tritonserver/Dockerfile:5` | none (confirmed correct) | The bump does address the reported `libcudart.so.12` failure. | Verified on disk: line 7 `FROM ${TRITON_SERVER_IMAGE} AS triton_source`, line 9 `FROM ${DYNAMO_BASE_IMAGE} AS dynamo_base`, line 11 `COPY --from=triton_source /opt/tritonserver /opt/tritonserver`, lines 12-13 copy `/usr/local/dcgm` and `/lib/x86_64-linux-gnu/libdcgm*.so*`, line 26 `RUN uv pip install /opt/tritonserver/python/triton*.whl`. `container/context.yaml:12-15` gives the `dynamo` framework **exactly one** device key, `cuda13.0`, at `base_image_tag: 25.11-cuda13.0-devel-ubuntu24.04`. A CUDA-12 tree copied into a CUDA-13-only base is precisely the reported failure, and it is invisible to `docker build`. |
| 3 | `change.md:109` | nit | The v2.63.0 "subset of `nvcr.io/nvidia/tritonserver:25.11-py3`" quotation is truncated: the sentence continues "**ARM container image**", and this Dockerfile copies `/lib/x86_64-linux-gnu/...`. | I re-fetched the release body. Line 143 reads verbatim: ``* This package is a subset of `nvcr.io/nvidia/tritonserver:25.11-py3` ARM container image assets it.`` and sits under `<summary><h2>Jetson AGX Systems Support</h2>` (line 137). `change.md` quotes up to the backtick and stops, so the ARM scoping is dropped from a load-bearing citation. **Not blocking**: the validator caught this independently and recorded it at `change-validation.md:72-80`, so the packet set as a whole is honest, and the claim survives on the substitutes in finding 4. |
| 4 | `change-validation.md:82-99` | none (confirmed, and stronger than described) | The CUDA-13 claim holds on the x86 path independently of the ARM-scoped sentence. | Re-fetched myself. `docs/introduction/compatibility.md@v2.63.0` gives per-tag CUDA, arch-independent: `25.11-trtllm-python-py3` and `25.11-vllm-python-py3` → CUDA `13.0.2.006`; the old pin's rows `25.01-*-python-py3` → CUDA `12.8.0.038`; the boundary is visible in the same table at `25.07` → `12.9.0.043`, `25.08` → `13.0.1.012`. Independently: `build.py@r25.01:1550` copies `libcudart.so.12`, `build.py@r25.11:1582` copies `libcudart.so.13`, with `libcupti`/`libnvJitLink`/`libcublas`/`libcublasLt` moving `.12→.13` in lockstep. Both halves of the diagnosis — the old pin really was CUDA 12, the new one is CUDA 13 — are attested. |
| 5 | `change-validation.md:96-99` | nit | The `build.py` soname evidence carries a scoping caveat the validator did not state. Those `COPY --from=min_container` lines live in `add_cpu_libs_to_linux_dockerfile`, which `build.py@r25.11:1428` invokes only in the **`else` branch of `if enable_gpu:`** — i.e. the CPU-only image. | The validator noted the arch-parameterization (`cuda_arch = "sbsa" if aarch64 else "x86_64"`) but not the `enable_gpu=False` scoping. The evidence remains probative — the whole r25.11 build tree targets CUDA-13 sonames where r25.01 targeted CUDA-12 — and the GPU path is separately bound: `build.py@r25.11:1710` sets `base_image = "nvcr.io/nvidia/tritonserver:{upstream_container_version}-py3-min"`, so the `-py3` image inherits its CUDA from the min container of the same monthly train. The **compatibility.md** table in finding 4 is the cleaner citation and is not subject to this caveat. |
| 6 | `plan.md:213-216`, `change.md:69-73` | nit | The "`25.12-py3` and later are CUDA 13.1" premise for choosing `25.11` as ceiling is well-founded but not airtight, because upstream's compatibility matrix has **no plain `-py3` section** — only backend-suffixed variants. | `compatibility.md@v2.64.0` gives `25.12-vllm-python-py3` → CUDA `13.1.0.036` (vs `25.11` → `13.0.2.006`), and v2.64.0's AGX section states CUDA `13.1`. But the same table's `25.12-trtllm-python-py3` row still reads `13.0.2.006`, because the TRT-LLM container lags a train (v2.63.0 states it "is built from the 25.10 image `…:25.10-py3-min`"). So per-tag CUDA is not uniform across `-py3` variants. **This argues *for* the conservative choice, not against it** — it is an additional reason not to reach past `25.11`. |
| 7 | `examples/backends/tritonserver/Dockerfile:5` | none (confirmed) | `25.11-py3` is a defensible deviation from the caller's literal `25.08-py3`. | The caller wrote "25.08-py3 or newer", so `25.11` is inside the ask, not outside it. The three grounds check out: CUDA minor matches the base's `13.0` exactly (finding 4); same NGC monthly train as the base, which matters because lines 12-13 copy DCGM (`dcgm_version` moves `3.3.6`@r25.01 → `4.4.0-1`@r25.11, so the DCGM generation genuinely differs by train); and CPython minor matches — `build.py@r25.11:82` `"rhel_py_version": "3.12.3"` and compatibility.md's `Python 3.12.3` against `container/context.yaml:36` `python_version: "3.12"`, so the wheel at line 26 is cp312 into a cp312 interpreter. Both `25.08` and `25.11` satisfy the ask; `change.md:83-87` records `25.08-py3` as the documented minimum and fallback, so a reviewer preferring the caller's literal number can swap one string with no re-derivation. That is the right way to handle a preference the reviewer may reasonably overturn. |
| 8 | `change.diff` | none (confirmed) | Scope is genuinely one line; the plan's non-goals held. | `git diff main...HEAD --stat` → `1 file changed, 1 insertion(+), 1 deletion(-)`, `--name-status` → `M examples/backends/tritonserver/Dockerfile` only. No `container/context.yaml`, no `Makefile`, no CI, no `src/`/`model_repo/`/`launch/`. |
| 9 | repo-wide | none (confirmed) | No necessary accompanying change was missed — there is **no** README or doc restating the old tag. | I re-ran the greps unpiped. `grep -rIn --exclude-dir=.git -E '25\.01'` over yaml/yml/md/mdx/Dockerfile\*/sh/py/toml → **exit 1**, no hits anywhere in the tree. `grep -rIn 'TRITON_SERVER_IMAGE'` → only Dockerfile lines 5 and 7. `grep -rIn -E 'tritonserver:[0-9]{2}\.[0-9]{2}'` across the whole repo → the changed line, and nothing else. `README.md`'s only CUDA mention is line 28, "NVIDIA GPU with CUDA support" — no tag. One version owner, one line. This is a complete fix, not a half-fix. |
| 10 | `change.md:133-152`, `change-validation.md:129-154` | none (confirmed honest) | The packets state the coverage gap plainly and do not imply more than exists. | Both say in plain words that the image was **not built, pulled, or inspected**; that `docker` is absent from `PATH`; that `api.ngc.nvidia.com` returns `CONNECT tunnel failed, response 403` so **it is unconfirmed that `nvcr.io/nvidia/tritonserver:25.11-py3` exists as a published tag**; and both distinguish "upstream release metadata describes what a release *is*" from "only the registry proves what was *published*." `change.md:172-175` explicitly declines to offer the green linter as runtime proof, quoting the dependency-upgrade template's own warning. `change-validation.md:148-150` explicitly declines to pad the packet with an unrelated vLLM smoke. I found no overstatement anywhere except the truncated quotation in finding 3. |
| 11 | `change-validation.md:113-121` | none (confirmed) | `N/A` for Recipe 06 was correctly *declined* rather than taken as an easy out. | `compute-env.md:31-33` offers `N/A` "when CI owns the Dockerfile/image proof". I verified CI does not: `grep -rniE 'triton' .github/workflows/` → **exit 1** across 45 workflow files; `.github/filters.yaml` has path filters at lines 215/223/232/244 for `examples/backends/{vllm,sglang,trtllm,sample}` and **no** `tritonserver` entry. Claiming `N/A` would have been a false assertion that some downstream system produces the missing proof. Declining it was the honest call. |
| 12 | `examples/backends/tritonserver/Makefile:29` | nit (out of scope, correctly deferred) | A **second, unpinned** owner of the effective Triton version survives this change, and the README documents that path. | Line 29 is `cd /tmp && git clone https://github.com/triton-inference-server/server.git` with no `--branch`/tag, feeding `build.py` at line 38. `make all` therefore tracks upstream `main` — today a 26.x/CUDA-13.1 tree — and installs it alongside the same CUDA-13.0 base. Genuinely a different defect from the container ABI mismatch, correctly out of scope for DYN-3697, and correctly disclosed in both `plan.md:33-42` and `change.md:182-188` so it reads as a decision rather than an oversight. Worth a follow-up ticket. |

### Evidence-table audit

`node /workflow/agents/customs/evidence.ts report` → `## Evidence [2/3 validated · 1 sandbox denied]`, `customs: cleared`. Every recipe the plan named in its `validation-recipes` block appears; none is `missing`.

- **`01-python-lint` — `validated`.** Cited log `validation/logs/2026-08-03T14-46-18.182Z-pre-commit-4e71.log` shows `pre-commit` genuinely ran against the changed file: `codespell`, case-conflict, merge-conflict, shebang-executable, mixed-line-ending, trailing-whitespace and the pytest-marker report all `Passed`; Python/Rust/JSON/YAML hooks `(no files to check) Skipped`, correct for a Dockerfile-only diff. Real execution, real pass. It is a weak exercise of the change and **both packets say so explicitly** (`change-validation.md:22-24`), so there is no contradiction to catch.
- **`05-code-inspection` — `validated`.** Cited log `validation/logs/2026-08-03T14-49-42.881Z-bash-6b6e.log` contains actual `gh api` output — the r25.11 `libcudart.so.13` COPY block and the full r25.01 `libcudart.so.12`/`libcublas*.so.12` reference list, with `Q1 exit=0`. The commands executed and exercised the claim under review.
- **`06-dockerfile-build` — `sandbox denied`, not `validated`.** Log shows `spawn docker ENOENT`. Correctly dispositioned; no green row is contradicted by its own evidence.

**No new test infrastructure** is introduced by this change (`checklist: new-infrastructure-exercise-cited` — satisfied by that observation). **Commit-on-real-branch**: HEAD is `8f41c2cbeeeb992f04e1bd7282e73221abc17b59` on branch `fix/triton-cuda13-abi-mismatch--1fdf61042fb3` (not the upstream default `main`), matching the SHA in `change.md:6`; `git status --porcelain` is empty; DCO trailer `svc-glamr@nvidia.com` matches the author. **Compute pathway**: `compute-env.md` was respected — no remote compute, SSH, or Slurm was attempted, and no GPU-requiring recipe was selected (correctly: nothing on the vLLM path changed, so a smoke would have been tautological).

---

## Requested changes

None that the agency can act on. The single deficit is unobtainable in this sandbox and
is not obtainable from CI either, so no further iteration is warranted; the run should
publish as `[blocked]` / *validation incomplete*.

For the **human** reviewer, one check closes the loop:

```bash
docker pull nvcr.io/nvidia/tritonserver:25.11-py3          # also confirms the tag is published
cd examples/backends/tritonserver && docker build -t dynamo-triton:latest .
docker run --rm dynamo-triton:latest ldd /opt/tritonserver/bin/tritonserver | grep -i cudart
```

Expect `libcudart.so.13`, resolved, with no `not found` lines. If `25.11-py3` turns out
not to be published, `25.08-py3` — the caller's own floor and the first CUDA-13 Triton
train — is the documented fallback and is a one-string swap on the same line.

Two items worth a separate ticket rather than this MR: the unpinned Makefile clone
(finding 12) and the absence of any CI job that builds this image (finding 11), which is
why this class of drift went unnoticed since the pin was introduced.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 63352be4-2cfc-48ac-a081-7bf6a6e2489b

📥 Commits

Reviewing files that changed from the base of the PR and between 7054447 and 8f41c2c.

📒 Files selected for processing (1)

• examples/backends/tritonserver/Dockerfile

---

Walkthrough

The Triton Server Dockerfile now defaults to the 25.11-py3 base image instead of 25.01-py3.

Changes

Triton Server image update

Layer / File(s)	Summary
Update the default Triton Server image examples/backends/tritonserver/Dockerfile	The TRITON_SERVER_IMAGE build argument now defaults to 25.11-py3.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the Triton example image update to a CUDA 13-compatible tag.
Description check	✅ Passed	The description includes all required sections and clearly explains the change, rationale, review path, validation status, and issue linkage.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[glamr-agent]

CI result: passed.

Observed terminal state on the PR's check rollup: 15 checks passed, 6 skipped, none failed
or pending.

Passing: pre-commit, DCO, dco-comment, copyright-checks, codeowners, label,
changed-files, ok-to-test, pr_reminder, Validate PR title and add label,
Check for broken markdown links, lychee, CodeRabbit, Devin Review,
pre-merge-status-check.

Skipped: Fern Configuration Check, Fern Broken Links Check, operator, rust-clippy,
rust-tests, snapshot — correct for a diff that touches one Dockerfile line and no Rust,
Python, docs, or operator source.

Note for the reviewer: no check in this set builds the Triton example image. The green
rollup does not stand in for the container ABI proof described in the evidence comment —
.github/workflows/ contains no Triton reference and .github/filters.yaml has no
examples/backends/tritonserver path filter, so no job here exercises the change. Full CI
on this repository additionally requires a maintainer to comment /ok to test <sha>.

[dagil-nvidia]

Closing in favour of #12577, which is the fuller fix for the same bug (NVBug 6541824 / DYN-3697).

This PR bumps TRITON_SERVER_IMAGE only. #12577 also fixes the DCGM library copy: the old image's triton_bindings linked against libdcgm.so.3 and libcudart.so.12, and Triton 25.10 ships libdcgm.so.4 under the merged-usr path, so the ARG bump alone leaves the container failing to start for a second reason.

#12577 is also authored by the bug's owner and pins a specific CUDA 13.0.2 tag rather than a floor.

No fault in the run: the dispatch brief specified only the ARG bump, so this built exactly what was asked for. Recording it so the miss is attributed to the brief, not the agent.