chore(container): bump tensorrt-llm pin from 1.3.0rc11 to 1.3.0rc13

[yifjiang]

Summary

Bump the default tensorrt-llm PyPI pin in container/context.yaml from 1.3.0rc11 (tagged 2026-04-07) to 1.3.0rc13 (tagged 2026-04-26). rc13 is the first published release tag that includes NVIDIA/TensorRT-LLM#12976 ("Fix compute token accounting for KV cache reuse with context chunking"), the C++ admission fix for NVIDIA/TensorRT-LLM#13318.

Tag-by-tag verification

Verified by checking cpp/tensorrt_llm/batch_manager/microBatchScheduler.cpp on each tag tree:

Tag	Date	reuse_adjusted_compute count	Has admission fix?
v1.3.0rc11 (4e69c14f7)	2026-04-07	0	❌
v1.3.0rc12 (61cef212a)	2026-04-16	0	❌ (rc12 was cut 2 days before #12976 merged)
v1.3.0rc13 (b9ce4b69d)	2026-04-26	10 (1 static helper def + 9 call sites)	✅

So rc11 → rc13 is the smallest possible jump that brings in the upstream fix. rc12 is intentionally skipped — it does not include the fix.

Why this matters

Without the C++ admission fix, dynamo + TRT-LLM under workloads that combine enable_chunked_prefill: true, enable_block_reuse: true, and tight max_num_tokens can hit:

AssertionError: total_num_tokens (N) should be less than or equal to max_num_tokens (M)

(reproducible on rc11 with Qwen3-4B / GB200 + multi-turn-chat replay at qps≥12; for the same workload, an image built with the rc13 wheel does not crash). The bug is C++-side: chunked-prefill non-first-chunk admissions were over-admitted relative to max_num_tokens because the scheduler did not subtract the discounted reuse cost correctly. PR #12976 fixes the math at admission time.

Files changed

• container/context.yaml — pip_wheel: tensorrt-llm==1.3.0rc11 → rc13 and github_trtllm_commit: v1.3.0rc11 → v1.3.0rc13 (the second key drives the URL we curl install_tensorrt.sh from in container/templates/trtllm_framework.Dockerfile:129).
• container/deps/requirements.common.txt — comment-only update on the transformers constraint line (rc11 → rc13; same ==4.57.3 constraint applies, no actual transformers pin change).
• docs/reference/support-matrix.md — bump the **main (ToT)** row's TRT-LLM column from 1.3.0rc11 → 1.3.0rc13. The historical v1.1.0-dev.3 row stays at 1.3.0rc11 because that release shipped with rc11 — that is a fact about the past.

docs/reference/release-artifacts.md is intentionally not changed: every 1.3.0rc11 reference in that file is in the v1.1.0-dev.3 release-history section (lines 169 / 587 / 594), describing what that past release shipped — historical, not ToT.

Verification done locally

• ARM64 (GB200) container built with this change pulls tensorrt-llm==1.3.0rc13 from pypi.nvidia.com cleanly. has_trtllm_context: "0" path; no custom wheel.
• The shipped libtensorrt_llm.so matches the rc13 release wheel (provenance confirms fix(container): make vLLM FlashInfer cubin dirs writable (#12770) #12976's C++ admission fix is compiled in; the helper is static and inlined by the optimizer so symbol-grep is unreliable, but byte-identity against the rc13 wheel establishes presence).
• A separate harsh-load repro on the resulting image is in progress.

Test plan

• CI: trtllm-pipeline (covers TRT-LLM container-build + smoke test against rc13)
• CI: pre-commit (no Python changes; should be a no-op)
• Manual: pull the resulting dynamo-trtllm container, run a known-good aggregated-serving smoke test on Qwen3-4B / 1× GB200, confirm /v1/chat/completions returns 200.
• Manual: rerun the issue docs: fix dead links and anchors left by the docs path re-alignment #13318 harsh-config repro (multi-prefix multi-turn at qps≈18 with max_num_tokens=4096) and confirm zero AssertionError: total_num_tokens > max_num_tokens events.

Related

• Upstream issue: [Bug]: Scheduler deadlock on main + #12976 + #13029: AssertionError total_num_tokens > max_num_tokens in _prepare_tp_inputs under KV offload + chunked prefill permanently hangs the event loop NVIDIA/TensorRT-LLM#13318
• Upstream fix: [None][fix] Fix compute token accounting for KV cache reuse with context chunking NVIDIA/TensorRT-LLM#12976 (already merged to TRT-LLM main, 2026-04-18)
• Backport-to-rc13: included in v1.3.0rc13 release tag (2026-04-26)
• Upstream PR for the Python re-validation safety net (still OPEN against TRT-LLM main): [None][fix] Cherry-pick Python-side remaining_budget guard for KV reuse budget overflow NVIDIA/TensorRT-LLM#13601 — defense-in-depth on top of fix(container): make vLLM FlashInfer cubin dirs writable (#12770) #12976; not required for this dynamo-side bump

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-d9701c6a-1e4c-4dc2-922a-8f97d81413be.docs.buildwithfern.com/dynamo/dev

[yifjiang]

E2E test on the rc13-built image — done

Built nvcr.io/goirlvsxnepa/llm_nim/dynamo-trtllm:head-tot-rc13-2026-04-29 from this branch (ARM64 / GB200) and ran the issue #13318 repro. Clean run.

Setup

• Test node: dlcluster gb-nvl-057-compute03, 1× GB200, fresh allocation (no other tenant on GPU 0)
• Engine config matching issue docs: fix dead links and anchors left by the docs path re-alignment #13318: max_num_tokens=4096, enable_chunked_prefill=true, enable_block_reuse=true, enable_partial_reuse=true, V1 scheduler (use_kv_cache_manager_v2=false, use_python_scheduler=false), host_cache_size=50 GiB, free_gpu_memory_fraction=0.80
• Trace: 6022 requests / 800 sessions / 6-9 turns each, last-turn prompt-token median 3065, prefix-share 0.355
• Replay: warmup qps=8 × 6 rounds × 30 s + trigger qps=18 × 8 rounds × 30 s, request-timeout=30 s

Verification

import tensorrt_llm; tensorrt_llm.__version__ == "1.3.0rc13"     # confirmed inside the built image
[TRT-LLM] log line at startup: "TensorRT LLM version: 1.3.0rc13"  # confirmed

Smoke test

curl -X POST http://<node>:8088/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-4B","messages":[{"role":"user","content":"Say hello in one word."}],"max_tokens":12}'
# → 200 OK, content "<think>\nOkay, the user asked me to say hello in"

Sustained-load run (issue #13318 trigger phase)

[warmup][r0..r5]  240/240 OK each, achieved_qps≈7.85, p50 ~1.2 s
[trigger][r0]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=16.26 p50=10.80s   (graph-warmup spike)
[trigger][r1]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.60 p50=2.67s
[trigger][r2]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.62 p50=2.05s
[trigger][r3]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.60 p50=1.91s
[trigger][r4]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.59 p50=1.68s
[trigger][r5]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.59 p50=1.63s
[trigger][r6]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.58 p50=1.89s
[trigger][r7]     sent=540 ok=540 fail=0 timeout=0 achieved_qps=17.58 p50=1.67s
exit 0

Decoder-log diagnostics

Counter	Value
AssertionError: total_num_tokens > max_num_tokens	0
MNT overflow:	0 (only present if PR #13601's diagnostic logger were applied; not in rc13)
tensorrt_llm.executor.utils.RequestError	0
/health after run	healthy, worker registered
Total OK / attempted (warmup + trigger)	5760 / 5760 = 100%

For comparison: an rc11-based image (head-health-cuda-v4-benchy-rc11 overlay) crashes on the same trace + same engine config + same qps=18 with AssertionError: total_num_tokens (4573) should be less than or equal to max_num_tokens (4096) and SCHEDULER_HANG_DETECTED at trigger r6, exit code 2 — that data is in docs/v4_overlay_negative_result.md of the orchestrator-side repo. The rc13 image's clean run on the identical workload is the empirical confirmation that PR NVIDIA/TensorRT-LLM#12976 (which lands in rc13) is the load-bearing fix and that bumping the dynamo wheel pin to rc13 closes issue #13318 for users on the default has_trtllm_context: "0" path.

Test plan boxes

• Manual: tensorrt-llm==1.3.0rc13 is what gets installed (verified at runtime)
• Manual: aggregated-serving smoke test on Qwen3-4B / 1× GB200 returns 200
• Manual: issue docs: fix dead links and anchors left by the docs path re-alignment #13318 harsh-config replay (qps=18 trigger, 4320 attempted requests at trigger phase) — zero AssertionError events, zero RequestErrors, server stays healthy
• CI: trtllm-pipeline (will run when this PR is taken out of draft)
• CI: pre-commit (will run when out of draft; expected to pass — no Python or template changes)

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.