Skip to content

[None][chore] AutoDeploy add multi stream moe pass to default.yaml #9430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
suyoggupta merged 4 commits into from
Nov 25, 2025
Merged

[None][chore] AutoDeploy add multi stream moe pass to default.yaml #9430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
12a217d
add multi stream moe pass to default.yaml
suyoggupta Nov 25, 2025
e486bb8
enable multi-stream moe in acc test
suyoggupta Nov 25, 2025
706dc64
enable multi-stream moe in acc test
suyoggupta Nov 25, 2025
9363f43
Merge branch 'main' into sg/hot-fix
suyoggupta Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions tensorrt_llm/_torch/auto_deploy/config/default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,9 @@ transforms:
############################################################################################
fuse_causal_conv_activation:
stage: compile
multi_stream_moe:
stage: compile
enabled: false
compile_model:
stage: compile
run_per_gm: false
Expand Down
7 changes: 7 additions & 0 deletions tests/integration/defs/accuracy/test_llm_api_autodeploy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,10 @@ def get_default_kwargs(self):
"sharding_source": ['factory', 'heuristic'],
"sharding_dims": ['ep', 'bmm'],
},
"multi_stream_moe": {
"stage": "compile",
"enabled": True,
},
# NOTE: some accuracy benchmarks may require fp32 precision for mamba cache
# "insert_cached_ssm_attention": {
# "cache_config": {
Expand All @@ -201,6 +205,9 @@ def get_default_sampling_params(self):
@pytest.mark.skip_less_device_memory(32000)
def test_bf16(self):
kwargs = self.get_default_kwargs()
# TODO: multi-stream MOE seems to increase the memory usage
kwargs["max_batch_size"] = 32
kwargs["free_mem_ratio"] = 0.5
sampling_params = self.get_default_sampling_params()
with AutoDeployLLM(model=self.MODEL_PATH_BF16,
tokenizer=self.MODEL_PATH_BF16,
Expand Down