fix(generator): preserve Sweeper KV capacity [6738325] by jasonqinzhou · Pull Request #127 · ai-dynamo/aisimulate

jasonqinzhou · 2026-09-08T18:08:12Z

Summary

accept Sweeper recommendation candidates whose KV capacity uses backend defaults or a fixed block count
preserve default-capacity semantics for vLLM, SGLang, and TensorRT-LLM
render fixed capacity as --num-gpu-blocks-override for vLLM and as an exact token capacity for SGLang/TensorRT-LLM
reject ambiguous candidates that specify both a memory fraction and fixed blocks

Landing sequence

canonical base: main
clean source base: e08f0c3f33bcd725669fbe105085a363eeadd4a9
after the main change merges, a fresh cherry-pick PR will target release/0.12.0

Validation

generator unit suite plus release backend-contract goldens: 350 passed, 4 skipped
TensorRT-LLM fixed-capacity artifact probes: rc20 and rc23 both render max_tokens: 16384 without a memory-fraction fallback
clean no-commit merge with PR fix(generator): render SGLang candidate token limits [6738246] #128; combined generator/release-contract suite: 352 passed, 4 skipped
changed-file Ruff check and format check: passed
git diff --check: passed
full exact-head CI: run 34266444811 (in progress)

copy-pr-bot · 2026-09-08T18:08:15Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

jasonqinzhou · 2026-09-08T18:29:20Z

Release propagation note: this is an urgent release/0.12.0 hotfix, while the same bridge code remains unfixed on current main. Keep this PR release-targeted, but forward-port the final reviewed change to main after the release fix lands so the defect does not recur in the next main-derived release.

simone-chen

Should we remove the nvbug link from the PR description?

jasonqinzhou · 2026-09-08T18:41:35Z

Exact-head CI note: I attempted the repository standard pull-request/127 shadow branch to trigger ci.yml for b456ea7, but the live organization rules rejected ref creation with GH013. Local exact-head validation is complete (348 passed, 4 skipped; Ruff and diff checks green), and the combined tree with #128 passes 350 tests. A maintainer with ruleset bypass should run the full repository CI matrix before merge; this PR is ready for code review now.

jasonqinzhou · 2026-09-08T18:42:59Z

@simone-chen Removed the internal NVBug URL from the PR description. The release-fix identifier remains in the title for traceability.

Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>

jasonqinzhou · 2026-09-08T19:05:43Z

Retargeted to main as requested. Because release/0.12.0 and main have diverged, the head was safely rebuilt from current main and updated with --force-with-lease; the resulting diff is limited to this fix. Full CI is running on exact head d2c11d29bfd3286fee72932f6b4a4d318f8cff93.

* fix(generator): preserve Sweeper KV capacity [6738325] * fix(generator): preserve fixed capacity for TRT-LLM rc23 --------- (cherry picked from commit 58a5e18) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>

* fix(ci): align public Rust schema contract (#78) (#79) (cherry picked from commit 01d104e) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(compliance): address OSRB filing requirements (#81) (#92) * fix(compliance): remediate OSRB filing requirements * fix(ci): stop archiving wheels in GitHub Actions * feat(ci): stage wheels in Artifactory * fix(ci): run wheel staging with bash * fix(codeowners): classify root dockerignore * docs(legal): reproduce Transformers license * fix(compliance): preserve provenance and legal payloads * fix(ci): validate wheels before staging * chore(codeowners): route compliance tests --------- (cherry picked from commit aab4635) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): serialize attention backend before save (#88) (#93) (cherry picked from commit 99d1d22) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): preserve Sweeper KV capacity [6738325] (#127) (#133) * fix(generator): preserve Sweeper KV capacity [6738325] * fix(generator): preserve fixed capacity for TRT-LLM rc23 --------- (cherry picked from commit 58a5e18) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): render SGLang candidate token limits [6738246] (#128) (#134) * fix(generator): render SGLang candidate token limits [6738246] * refactor(generator): isolate SGLang token lowering --------- (cherry picked from commit ffcb657) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(compliance): complete third-party license notices (#95) (#148) (cherry picked from commit 4b1b5dc) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> * chore: canonicalize repository governance files (#106) (#149) * chore: canonicalize repository governance files * fix(docs): keep packaged governance links resolvable --------- (cherry picked from commit d83e056) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * Harden SPDX checks and correct 50 Jinja license headers (#146) (#150) * ci: validate complete SPDX license identifiers * fix: make Jinja SPDX identifiers machine-readable * docs: add literal DCO sign-off command * docs: explain wheel-local legal file copies * docs: align replay parity reference with Apache-2.0 * fix: require closing SPDX block delimiters * fix: reject dangling SPDX comment delimiters --------- (cherry picked from commit 82de117) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(cli): preserve whole-model FPM FMHA identity [6745146] (#151) (#154) (cherry picked from commit b6b0a38) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): omit unspecified SGLang KV-cache dtype [6745807] (#152) (#155) (cherry picked from commit e7ea5d9) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(packaging): backport manylinux 2.28 wheel repair (#205) * fix(packaging): repair Linux wheels for manylinux 2.28 (#202) * fix(packaging): repair Linux wheels for manylinux 2.28 Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> * fix(ci): use bundled Python in manylinux jobs Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * test(packaging): verify manylinux repair behavior and pin Rust Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> --------- Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(ci): build staged wheels for glibc 2.28 --------- Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * docs: document Dynamo Planner prerequisites for AIS 0.12 Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> --------- Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Co-authored-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>

#291) * fix(ci): align public Rust schema contract (#78) (#79) (cherry picked from commit 01d104e) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(compliance): address OSRB filing requirements (#81) (#92) * fix(compliance): remediate OSRB filing requirements * fix(ci): stop archiving wheels in GitHub Actions * feat(ci): stage wheels in Artifactory * fix(ci): run wheel staging with bash * fix(codeowners): classify root dockerignore * docs(legal): reproduce Transformers license * fix(compliance): preserve provenance and legal payloads * fix(ci): validate wheels before staging * chore(codeowners): route compliance tests --------- (cherry picked from commit aab4635) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): serialize attention backend before save (#88) (#93) (cherry picked from commit 99d1d22) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): preserve Sweeper KV capacity [6738325] (#127) (#133) * fix(generator): preserve Sweeper KV capacity [6738325] * fix(generator): preserve fixed capacity for TRT-LLM rc23 --------- (cherry picked from commit 58a5e18) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): render SGLang candidate token limits [6738246] (#128) (#134) * fix(generator): render SGLang candidate token limits [6738246] * refactor(generator): isolate SGLang token lowering --------- (cherry picked from commit ffcb657) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(compliance): complete third-party license notices (#95) (#148) (cherry picked from commit 4b1b5dc) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> * chore: canonicalize repository governance files (#106) (#149) * chore: canonicalize repository governance files * fix(docs): keep packaged governance links resolvable --------- (cherry picked from commit d83e056) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * Harden SPDX checks and correct 50 Jinja license headers (#146) (#150) * ci: validate complete SPDX license identifiers * fix: make Jinja SPDX identifiers machine-readable * docs: add literal DCO sign-off command * docs: explain wheel-local legal file copies * docs: align replay parity reference with Apache-2.0 * fix: require closing SPDX block delimiters * fix: reject dangling SPDX comment delimiters --------- (cherry picked from commit 82de117) Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(cli): preserve whole-model FPM FMHA identity [6745146] (#151) (#154) (cherry picked from commit b6b0a38) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(generator): omit unspecified SGLang KV-cache dtype [6745807] (#152) (#155) (cherry picked from commit e7ea5d9) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(packaging): backport manylinux 2.28 wheel repair (#205) * fix(packaging): repair Linux wheels for manylinux 2.28 (#202) * fix(packaging): repair Linux wheels for manylinux 2.28 Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> * fix(ci): use bundled Python in manylinux jobs Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * test(packaging): verify manylinux repair behavior and pin Rust Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> --------- Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(ci): build staged wheels for glibc 2.28 --------- Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Co-authored-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(cli): resolve prediction backend versions on 0.12.0 Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(replay): apply authored versions before capacity estimation Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(replay): preserve null aliases and custom version roots Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(replay): align capacity systems root Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * style: format custom-root regression Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(epd): resolve omitted backend versions Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * test(epd): cover version lookup boundaries Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> * fix(replay): retain the canonical timing database root Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> --------- Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Co-authored-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>

#291) (#344) * fix(ci): align public Rust schema contract (#78) (#79) (cherry picked from commit 01d104e) * fix(compliance): address OSRB filing requirements (#81) (#92) * fix(compliance): remediate OSRB filing requirements * fix(ci): stop archiving wheels in GitHub Actions * feat(ci): stage wheels in Artifactory * fix(ci): run wheel staging with bash * fix(codeowners): classify root dockerignore * docs(legal): reproduce Transformers license * fix(compliance): preserve provenance and legal payloads * fix(ci): validate wheels before staging * chore(codeowners): route compliance tests --------- (cherry picked from commit aab4635) * fix(generator): serialize attention backend before save (#88) (#93) (cherry picked from commit 99d1d22) * fix(generator): preserve Sweeper KV capacity [6738325] (#127) (#133) * fix(generator): preserve Sweeper KV capacity [6738325] * fix(generator): preserve fixed capacity for TRT-LLM rc23 --------- (cherry picked from commit 58a5e18) * fix(generator): render SGLang candidate token limits [6738246] (#128) (#134) * fix(generator): render SGLang candidate token limits [6738246] * refactor(generator): isolate SGLang token lowering --------- (cherry picked from commit ffcb657) * fix(compliance): complete third-party license notices (#95) (#148) (cherry picked from commit 4b1b5dc) * chore: canonicalize repository governance files (#106) (#149) * chore: canonicalize repository governance files * fix(docs): keep packaged governance links resolvable --------- (cherry picked from commit d83e056) * Harden SPDX checks and correct 50 Jinja license headers (#146) (#150) * ci: validate complete SPDX license identifiers * fix: make Jinja SPDX identifiers machine-readable * docs: add literal DCO sign-off command * docs: explain wheel-local legal file copies * docs: align replay parity reference with Apache-2.0 * fix: require closing SPDX block delimiters * fix: reject dangling SPDX comment delimiters --------- (cherry picked from commit 82de117) * fix(cli): preserve whole-model FPM FMHA identity [6745146] (#151) (#154) (cherry picked from commit b6b0a38) * fix(generator): omit unspecified SGLang KV-cache dtype [6745807] (#152) (#155) (cherry picked from commit e7ea5d9) * fix(packaging): backport manylinux 2.28 wheel repair (#205) * fix(packaging): repair Linux wheels for manylinux 2.28 (#202) * fix(packaging): repair Linux wheels for manylinux 2.28 * fix(ci): use bundled Python in manylinux jobs * test(packaging): verify manylinux repair behavior and pin Rust --------- * fix(ci): build staged wheels for glibc 2.28 --------- * fix(cli): resolve prediction backend versions on 0.12.0 * fix(replay): apply authored versions before capacity estimation * fix(replay): preserve null aliases and custom version roots * fix(replay): align capacity systems root * style: format custom-root regression * fix(epd): resolve omitted backend versions * test(epd): cover version lookup boundaries * fix(replay): retain the canonical timing database root --------- (cherry picked from commit 7836e7b) Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com> Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Co-authored-by: Jason Zhou <jasonzho@nvidia.com> Co-authored-by: Harrison Saturley-Hall <hsaturleyhal@nvidia.com>

jasonqinzhou requested review from a team as code owners September 8, 2026 18:08

jasonqinzhou added the review-ready Ready for automated and human review label Sep 8, 2026

jasonqinzhou commented Sep 8, 2026

View reviewed changes

Comment thread ...aiconfigurator/generator/config/backend_templates/trtllm/extra_engine_args.1.3.0rc14.yaml.j2

jasonqinzhou mentioned this pull request Sep 8, 2026

fix(generator): render SGLang candidate token limits [6738246] #128

Merged

simone-chen approved these changes Sep 8, 2026

View reviewed changes

jasonqinzhou added 2 commits September 8, 2026 11:51

fix(generator): preserve Sweeper KV capacity [6738325]

a0cf02a

Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>

fix(generator): preserve fixed capacity for TRT-LLM rc23

d2c11d2

Signed-off-by: Jason Zhou (Engrg-Hardware 1) <jasonzho@nvidia.com>

jasonqinzhou mentioned this pull request Sep 8, 2026

fix(generator): preserve Sweeper KV capacity [6738325] #130

Closed

jasonqinzhou force-pushed the jasonzho/nvbug-6738325-kv-capacity-lowering branch from b456ea7 to d2c11d2 Compare September 8, 2026 19:04

jasonqinzhou requested review from a team as code owners September 8, 2026 19:04

jasonqinzhou changed the base branch from release/0.12.0 to main September 8, 2026 19:04

jasonqinzhou requested a review from simone-chen September 8, 2026 19:07

jasonqinzhou merged commit 58a5e18 into ai-dynamo:main Sep 8, 2026
16 of 17 checks passed

jasonqinzhou deleted the jasonzho/nvbug-6738325-kv-capacity-lowering branch September 8, 2026 19:26

This was referenced Sep 8, 2026

fix(generator): backport Sweeper KV capacity fix to 0.12.0 #133

Merged

fix(generator): backport SGLang token-limit fix to 0.12.0 #134

Merged

Navigation Menu

Repository navigation

fix(generator): preserve Sweeper KV capacity [6738325] - #127

fix(generator): preserve Sweeper KV capacity [6738325]#127
jasonqinzhou merged 2 commits into
ai-dynamo:mainfrom
jasonqinzhou:jasonzho/nvbug-6738325-kv-capacity-lowering

jasonqinzhou commented Sep 8, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 8, 2026

Uh oh!

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

simone-chen left a comment

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jasonqinzhou commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Landing sequence

Validation

Uh oh!

copy-pr-bot Bot commented Sep 8, 2026

Uh oh!

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

simone-chen left a comment

Choose a reason for hiding this comment

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

jasonqinzhou commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jasonqinzhou commented Sep 8, 2026 •

edited

Loading