Repository navigation
fix(container): update GLM-5.2 SGLang runtime dependencies - #15549
ynpandey-nv wants to merge 2 commits into
Conversation
Upgrade NATS and etcd, pin AnyIO 4.14.2, and remove the unused Nsight nic_sampler binary so a dev.2 rebuild can clear the dev.1 Critical findings. Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
|
/ok to test 1c1361e |
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
❌ Dynamo PR CI failed — run 36966577025 (attempt 1) on
|
| Framework | Build | 1-GPU amd64 | Multi-GPU amd64 |
|---|---|---|---|
| vLLM | ✅ 4 | ❌ 1 | ❌ 1 |
| SGLang | ❌ 1 ✅ 2 | ➖ | ➖ |
| TRT-LLM | ✅ 4 | ❌ 1 | ❌ 1 |
| Other | Jobs |
|---|---|
| dynamo-runtime | ⏹️ 1 |
| planner | ❌ 2 ✅ 3 |
Failure details
7 jobs failed with 3 distinct signatures: the 4 vLLM/TRT-LLM test jobs abort before running tests because the local action .github/actions/pytest-local is missing from the checkout; both planner CPU jobs hit a collection error (transformers cannot import ResolvedRevision from huggingface_hub); and the SGLang build's Compliance extract fails building NIXL after sccache could not load its S3 credentials. The cancelled dynamo-runtime build hit its 30-minute job timeout (not a fail-fast or manual cancel).
❌ vllm-runtime / Test cuda13.0, amd64: missing local action .github/actions/pytest-local
Job: vllm-runtime / Test cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 110720163503 -R ai-dynamo/dynamo --log-failed
✅ Sanity check passed
##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/_work/dynamo/dynamo/.github/actions/pytest-local'. Did you forget to run actions/checkout before running your local action?
The test step references the local composite action .github/actions/pytest-local, which does not exist in the checked-out tree, so no tests ran. The same error fails the other three framework test jobs below.
❌ vllm-runtime / Multi-GPU Test cuda13.0, amd64: missing local action .github/actions/pytest-local
Job: vllm-runtime / Multi-GPU Test cuda13.0, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 110720163490 -R ai-dynamo/dynamo --log-failed
##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/_work/dynamo/dynamo/.github/actions/pytest-local'. Did you forget to run actions/checkout before running your local action?
Same as above: the GPU test step cannot resolve .github/actions/pytest-local, so no tests ran.
❌ trtllm-runtime / Test cuda13.1, amd64: missing local action .github/actions/pytest-local
Job: trtllm-runtime / Test cuda13.1, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 110721340281 -R ai-dynamo/dynamo --log-failed
✅ Sanity check passed
##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/_work/dynamo/dynamo/.github/actions/pytest-local'. Did you forget to run actions/checkout before running your local action?
Same missing .github/actions/pytest-local action; no tests ran.
❌ trtllm-runtime / Multi-GPU Test cuda13.1, amd64: missing local action .github/actions/pytest-local
Job: trtllm-runtime / Multi-GPU Test cuda13.1, amd64 · Failed step: Run GPU tests (sequential) · Logs: gh run view --job 110721340246 -R ai-dynamo/dynamo --log-failed
##[error]Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/_work/dynamo/dynamo/.github/actions/pytest-local'. Did you forget to run actions/checkout before running your local action?
Same missing .github/actions/pytest-local action; no tests ran.
❌ planner / CPU Test cpu, amd64: ImportError ResolvedRevision from huggingface_hub
Job: planner / CPU Test cpu, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 110719693022 -R ai-dynamo/dynamo --log-failed
_ ERROR collecting components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py _
components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py:34: in <module>
from dynamo.replay.main import _engine_caps
/opt/dynamo/venv/lib/python3.12/site-packages/dynamo/mocker/utils/kv_cache.py:7: in <module>
from transformers import AutoConfig
/opt/dynamo/venv/lib/python3.12/site-packages/transformers/utils/hub.py:29: in <module>
from huggingface_hub import (
E ImportError: cannot import name 'ResolvedRevision' from 'huggingface_hub' (/opt/dynamo/venv/lib/python3.12/site-packages/huggingface_hub/__init__.py)
ERROR components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py - ImportError while importing test module
================== 894 passed, 15 skipped, 1 error in 23.49s ===================
All tests pass except a collection error in tests/offline/test_replay_adapter_fpm.py: the installed transformers imports ResolvedRevision from huggingface_hub, which the installed huggingface_hub does not provide (incompatible versions in the image's venv).
❌ planner / CPU Test cpu, arm64: ImportError ResolvedRevision from huggingface_hub
Job: planner / CPU Test cpu, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 110719692874 -R ai-dynamo/dynamo --log-failed
E ImportError: cannot import name 'ResolvedRevision' from 'huggingface_hub' (/opt/dynamo/venv/lib/python3.12/site-packages/huggingface_hub/__init__.py)
ERROR components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py - ImportError while importing test module
================== 893 passed, 16 skipped, 1 error in 31.41s ===================
Same transformers/huggingface_hub version mismatch on arm64, breaking collection of test_replay_adapter_fpm.py.
❌ sglang-runtime / Build multi-arch cuda13.0: NIXL meson "sccache nvcc cannot compile programs"
Job: sglang-runtime / Build multi-arch cuda13.0 · Failed step: Compliance extract · Logs: gh run view --job 110711612886 -R ai-dynamo/dynamo --log-failed
#145 0.176 sccache: error: Server startup failed: cache storage failed to read: Unexpected (temporary) at read => loading credential to sign http request
#145 0.176 called: reqsign::LoadCredential
#145 0.176 service: s3
#143 0.222 WARNING: sccache server failed to start, building without cache
#143 0.226 Cloning into 'nixl'...
#143 8.717 meson.build:194:4: ERROR: Compiler /usr/local/bin/sccache nvcc cannot compile programs.
ERROR: failed to build: failed to solve: process "/bin/sh -c export AWS_WEB_IDENTITY_TOKEN_FILE=/run/secrets/aws-token && ... meson setup build/ ... ninja && ninja install ..." did not complete successfully: exit code: 1
##[error]Process completed with exit code 1.
The image build itself succeeded; the Compliance extract step's Docker build failed in the wheel_builder NIXL stage. sccache could not start because it failed to load S3 credentials (reqsign::LoadCredential, "temporary"), and meson's nvcc check through the sccache nvcc wrapper then failed. This looks like an infrastructure (sccache/AWS credential) issue; a rerun of failed jobs is likely enough.
⏹️ dynamo-runtime / image / Build multi-arch cuda13.0: cancelled
Job: dynamo-runtime / image / Build multi-arch cuda13.0 · Not analyzed. Cancelled in Build and Push Image by the job timeout ("The job has exceeded the maximum execution time of 30m0s"), not a fail-fast or manual cancel.
For agents
{"pr": 15549, "run_id": 36966577025, "run_attempt": 1, "head_sha": "1c1361e98b3914f8301af8e026a93430519bf584",
"failures": [
{"job": "sglang-runtime / Build multi-arch cuda13.0", "job_id": 110711612886, "failed_step": "Compliance extract", "signature": "NIXL meson: sccache nvcc cannot compile programs (sccache S3 credential load failed)", "tests": [], "log_cmd": "gh run view --job 110711612886 -R ai-dynamo/dynamo --log-failed"},
{"job": "planner / CPU Test cpu, arm64", "job_id": 110719692874, "failed_step": "Run CPU-only tests (parallelized)", "signature": "ImportError: cannot import name 'ResolvedRevision' from 'huggingface_hub'", "tests": ["components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py"], "log_cmd": "gh run view --job 110719692874 -R ai-dynamo/dynamo --log-failed"},
{"job": "planner / CPU Test cpu, amd64", "job_id": 110719693022, "failed_step": "Run CPU-only tests (parallelized)", "signature": "ImportError: cannot import name 'ResolvedRevision' from 'huggingface_hub'", "tests": ["components/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.py"], "log_cmd": "gh run view --job 110719693022 -R ai-dynamo/dynamo --log-failed"},
{"job": "vllm-runtime / Multi-GPU Test cuda13.0, amd64", "job_id": 110720163490, "failed_step": "Run GPU tests (sequential)", "signature": "Can't find action.yml under .github/actions/pytest-local", "tests": [], "log_cmd": "gh run view --job 110720163490 -R ai-dynamo/dynamo --log-failed"},
{"job": "vllm-runtime / Test cuda13.0, amd64", "job_id": 110720163503, "failed_step": "Run CPU-only tests (parallelized)", "signature": "Can't find action.yml under .github/actions/pytest-local", "tests": [], "log_cmd": "gh run view --job 110720163503 -R ai-dynamo/dynamo --log-failed"},
{"job": "trtllm-runtime / Multi-GPU Test cuda13.1, amd64", "job_id": 110721340246, "failed_step": "Run GPU tests (sequential)", "signature": "Can't find action.yml under .github/actions/pytest-local", "tests": [], "log_cmd": "gh run view --job 110721340246 -R ai-dynamo/dynamo --log-failed"},
{"job": "trtllm-runtime / Test cuda13.1, amd64", "job_id": 110721340281, "failed_step": "Run CPU-only tests (parallelized)", "signature": "Can't find action.yml under .github/actions/pytest-local", "tests": [], "log_cmd": "gh run view --job 110721340281 -R ai-dynamo/dynamo --log-failed"}
]}Posted automatically by Devin for run 36966577025. Updated on every full-CI run of this PR.
dmitry-tokarev-nv
left a comment
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Approve at 1c1361e98b. One P2 is inline. One P3 and one open item are below.
- [P3] NATS 2.12 is end of life. The docs.nats.io support table lists
v2.12.15(2026-08-12) as the final 2.12 release.mainpinsv2.14.7, which shipsgo1.26.8andx/crypto v0.57.0. The base client passes the same probes on it. Please considerv2.14.7, so that a later flagged Go orx/cryptoversion in NATS can still get a fix. - Open: this review does not test the claim that the PR clears the 34 amd64 and 31 arm64 Critical rows. The rescan in the test plan is the test for it.
What I measured at 1c1361e.
- Archives: I fetched the NATS and etcd archives from the URLs in
dynamo_base.Dockerfile. All 8 match the releaseSHA256SUMS, and a changed copy fails. go version -m, on amd64 and arm64:nats-serverv2.12.15 hasgo1.25.12andx/crypto v0.55.0.etcd,etcdctlandetcdutlv3.5.34 havego1.26.8andx/crypto v0.55.0. The.debstill installs only/usr/bin/nats-server.- The shipped
sglang-runtime:1.3.0-glm-5.2-dev.1(amd64) holds the v2.10.28 and v3.5.30 release binaries, byte for byte. Its first 27 layers arelmsysorg/sglang:v0.5.14-cu130-runtime. - NATS server: two base
dynamo-runtimetests pass againstnats:2.12.15,nats:2.10.28andnats:2.14.7. They cover the object store and a stream with pull consumers. A throwaway KV probe throughNATSStorealso passes. With no server, all 3 fail. - Strict mode: 2.12.15 and 2.14.7 reject an unknown JetStream field, and 2.10.28 accepts it. So the client sent only known fields. The
nats:2.12.15binary is the release binary, byte for byte. - Kubernetes: the platform chart still runs
nats:2.10.21-alpine. This PR changes only the binary inside the images. - AnyIO: the CI build of this head logs
Found existing installation: anyio 4.14.1andSuccessfully installed anyio-4.14.2on both arches. The base image has no pip constraint. - PyPI: the trusted publisher
agronholm/anyio(publish.yml) uploaded the 4.14.2 wheel on 2026-07-12. Its sha256 is9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494. Its dependencies match 4.14.1. - Six installed packages require AnyIO: anthropic, httpcore (only with its
asyncioextra), httpx, openai, starlette and watchfiles. All six accept 4.14.2. nic_sampler: the image has one regular file, at/opt/nvidia/nsight-compute/2025.3.1/host/target-linux-x64/plugins/efa_metrics/nic_sampler. It is amd64 only and built withgo1.23.4.- The new
RUN, run as written in the dev.1 image, deletes it and keepsnsys. The guard alone fails on the untouched image. - Only
nsys-plugin.yamlrefers to it, as the executable of theefa_metricsplugin (nsys profile --enable=efa_metrics, AWS EFA counters). Nothing in this branch calls that plugin. - Scope:
context.yamlfeedsdynamo_basein every image. So the bump also changes the vLLM, TRT-LLM and Dynamo runtime images. - CI built the SGLang image for both arches at this head. No SGLang test job ran, so no CI test used this image.
| nats_version: v2.12.15 | ||
| # v3.5.34 ships Go 1.26.8 and golang.org/x/crypto v0.55.0. | ||
| etcd_version: v3.5.34 |
There was a problem hiding this comment.
[P2] The /legal files of the SGLang, vLLM and TRT-LLM runtime images still name NATS v2.10.28 and etcd v3.5.30. container/compliance/native_packages.yaml:128 and :143. Please set line 128 to v3.5.34 and line 143 to v2.12.15 in this PR, as main does in each NATS or etcd bump.
Measured at 1c1361e, with controls.
| Source | nats-server | etcd |
|---|---|---|
context.yaml at this head |
v2.12.15 |
v3.5.34 |
CI artifact vllm-runtime, amd64 and arm64 |
v2.10.28 |
v3.5.30 |
CI artifact sglang-runtime-efa, amd64 and arm64 |
v2.10.28 |
v3.5.30 |
CI artifact sglang-runtime, arm64 |
v2.10.28 |
v3.5.30 |
Native generator, head YAML, --native-image sglang-runtime |
v2.10.28 |
v3.5.30 |
| Same generator, the two lines bumped | v2.12.15 |
v3.5.34 |
| Same generator, a made-up image name | no rows | no rows |
The CI artifacts come from run 36966577025 on this head. The CSV rows and the CycloneDX purls agree, for example pkg:generic/nats-server@v2.10.28. The same run downloads nats-server-v2.12.15-*.deb and etcd-v3.5.34-*.tar.gz. The Compliance audit jobs pass because audit_image_sbom.py matches names only. On main at a88ef6df4c, #12917, #13016 and #14919 change both files together.
The change I ran through the generator:
- name: etcd
- version: v3.5.30
+ version: v3.5.34
@@
- name: nats-server
- version: v2.10.28
+ version: v2.12.15
Withdrawing my approval because new commits landed after it. I will review the new commits before I approve again.
Describe the NATS, etcd, AnyIO, and nic_sampler updates by version and package role only. Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
ynpandey-nv
changed the title
2 participants
Overview:
Updates the SGLang runtime dependencies used by
release/1.3.0-glm-5.2-dev.2.Details:
golang.org/x/cryptov0.55.0).golang.org/x/cryptov0.55.0).nic_samplerbinary, built with Go 1.23.4. The runtime only linksnsys.Where should the reviewer start?
container/context.yamlcontainer/deps/requirements.sglang.txtcontainer/templates/sglang_runtime.DockerfileRelated Issues
🚫 This PR is NOT linked to an issue:
Test plan
release/1.3.0-glm-5.2-dev.2after merge.golang.org/x/cryptov0.55.0.golang.org/x/cryptov0.55.0.nic_samplerpath remains andnsysis still linked.