perf: remove trtllm postprocessing workers from the args as post processing workers are not effective in dynamo - #12592
Merged
Conversation
Contributor
WalkthroughThe TensorRT-LLM worker removes ChangesTensorRT-LLM postprocess cleanup
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
This comment has been minimized.
This comment has been minimized.
krishung5
approved these changes
Aug 3, 2026
krishung5
left a comment
Contributor
There was a problem hiding this comment.
I see that the default value of num_postprocess_workers is 0, so stripping directly sounds good, thanks!
richardhuo-nv
force-pushed
the
rihuo/remove_pp_workers_and_warn
branch
from
August 3, 2026 23:11
b6eeb47 to
a4ba166
Compare
richardhuo-nv
force-pushed
the
rihuo/remove_pp_workers_and_warn
branch
from
August 3, 2026 23:11
a4ba166 to
e69e37d
Compare
tanmayv25
reviewed
Aug 4, 2026
tanmayv25
approved these changes
Aug 5, 2026
jthomson04
approved these changes
Aug 5, 2026
richardhuo-nv
enabled auto-merge (squash)
August 5, 2026 18:04
hhzhang16
added a commit
that referenced
this pull request
Aug 6, 2026
dyn-3691-extract-shared-target-pid-cuda-customstorage-operation-layer * 'main' of https://github.com/ai-dynamo/dynamo: (65 commits) fix(frontend): emit SGLang stream role once (#12741) docs(fern): promote v1.3.1 to current release (#12752) fix(docs): remove duplicate unscoped community-rail CSS rules (#12615) feat(operator): migrate CRD storage to v1beta1 (#11904) fix: synchronize self-benchmark capacity across DP ranks (#12021) chore(deps): bump dynamo-tokenizers to 1.8.0 (#12707) fix(frontend): preserve split UTF-8 characters (#12688) docs: align Kubernetes build selector with CLI (#12729) fix(frontend): preserve completion backend error status (#12706) fix(operator): replace snapshot pods after GMS restart (#11286) refactor(media): rename installer module, drop --packages per review fix(media): harden installer against three pre-redesign review findings fix(media): verify installs in a fresh interpreter; teach --pip-args= form test(serve): install test-time decoders at the validated bounds feat(media): explicit installer for additional media decoders docs(spica): correct kv_load_ratio support guidance (#12714) feat(operator): add experimental grove.forceScalingGroup for single-node components (#11772) fix(vllm): declare entry-stage engine_input_source in GLM-Image NIXL config (#12709) chore: bump trtllm to v1.3.0rc23 (#12532) perf: remove trtllm postprocessing workers from the args as post processing workers are not effective in dynamo (#12592) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_strip_postprocess_workers()helper inllm_worker.pythat removesnum_postprocess_workersfrom TRT-LLM engine args before the engine is initialized, and emits aWARNING-level log if the value was> 0._strip_postprocess_workers()after all engine arg sources (--extra-engine-argsYAML and--override-engine-argsJSON) have been merged, so it covers every code path.engine_argspassed toget_llm_enginewhen supplied via an--extra-engine-argsYAML file (the pattern used in thekimi-k2.5,deepseek-v32-fp4, andgpt-oss-120brecipes).Why: TRT-LLM's
num_postprocess_workersspawns workers for its own post-processing pipeline, which conflicts with Dynamo's response-streaming path. When the setting is forwarded to TRT-LLM it has no useful effect and can occupy extra cpus resources causing perf degradation.Validation
pytest -m unit components/src/dynamo/trtllm/tests/test_trtllm_unit.py -k "postprocess"— all 4 new tests pass.num_postprocess_workers: 4and is absent when the key is not set.Related Issues
🚫 This PR is NOT linked to an issue:
Summary by CodeRabbit