Skip to content

Add new FW (TRT) and precision support - #5

Merged
kimbochen merged 13 commits into
mainfrom
kepotdar-trt-init
Sep 5, 2025
Merged

Add new FW (TRT) and precision support#5
kimbochen merged 13 commits into
mainfrom
kepotdar-trt-init

Conversation

@kedarpotdar-nv

@kedarpotdar-nv kedarpotdar-nv commented Sep 5, 2025

Copy link
Copy Markdown
Collaborator

Overview

This PR adds TensorRT-LLM (TRT-LLM) as a new inference framework for LLaMA 70B benchmarking on NVIDIA H200 and B200 GPUs, alongside the existing vLLM framework. This enables direct performance comparison between vLLM and TRT-LLM on the same hardware.

Key Features

  • Multi-framework support: vLLM and TRT-LLM for LLaMA 70B
  • Precision support: FP8 (default) and FP4 (future-ready)
  • Unified plotting: All frameworks and hardware on single performance plots
  • Framework-specific Docker images: Prevents image conflicts between frameworks
  • Clean job naming: Clear identification in GitHub Actions UI

🔧 Core Workflow Updates

.github/workflows/benchmark-tmpl.yml

  • Added framework and precision as required inputs
  • Updated RESULT_FILENAME to include framework and precision
  • Modified result processing to pass framework and precision to process_result.py
  • Fixed job naming to prevent duplication
  • Added hardware extraction logic for proper result processing

.github/workflows/70b-tmpl.yml

  • Added bmk-h200-trt and bmk-b200-trt jobs for TRT-LLM
  • Configured TRT-LLM jobs with nvidia/tensorrt-llm Docker image
  • Set precision to empty string for FP8 (default)
  • Updated all jobs to include framework and precision inputs

🚀 New Benchmark Scripts

benchmarks/70b_h200_trt_slurm.sh and benchmarks/70b_b200_trt_slurm.sh

  • TRT-LLM server setup for H200 using mpirun trtllm-serve
  • Inline llama-config.yml configuration
  • Client benchmarking with benchmark_serving.py

🔄 Launcher Script Updates

Updated SLURM Launchers
runners/launch_h200-nv.sh
runners/launch_h200-cw.sh
runners/launch_h200-nb.sh
runners/launch_b200-nv.sh
Key improvements:

  • Framework-specific SQSH file naming to prevent Docker image conflicts
  • Dynamic script selection based on framework (VLLM/SGLang use base scripts, TRT uses _trt scripts)
  • Proper MODEL_CODE environment variable passing to containers
  • Framework reset logic for VLLM and SGLang to use default script names

📊 Result Processing & Visualization

utils/process_result.py

  • Added framework and precision command-line arguments
  • Updated output data structure to include framework and precision
  • Default precision handling (empty string → 'fp8')

utils/plot_perf.py

  • Added distinct colors for TRT-LLM results:
    h200-trt: dark green
    b200-trt: gray
  • Unified plotting: all frameworks and hardware on single plots
  • Updated plot titles and legend handling
  • Model-specific plot generation

🧪 Testing Configuration

.github/workflows/workflow-scheduler.yml

  • Commented out concurrency and schedule blocks for manual testing
  • Disabled DSR1 jobs as requested

kimbochen
kimbochen reviewed Sep 5, 2025
View reviewed changes

@kimbochen kimbochen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR. lgtm

@kimbochen
kimbochen merged commit 0ef8128 into main Sep 5, 2025
@kimbochen
kimbochen deleted the kepotdar-trt-init branch September 5, 2025 04:51
@claude-code-infmax claude-code-infmax Bot mentioned this pull request Jan 21, 2026
Merged
Oseltamivir added a commit that referenced this pull request Jun 23, 2026
@Oseltamivir
CollectiveX: per-job summary table + address PR review findings
b384171 
Add summarize.py (compact NCCL/DeepEP results table, printed at end of every job) and make it the result gate. Fix review findings: benchmark failures/skipped-deepep now fail the job instead of reporting green (#1); DeepEP nodes from SLURM_NNODES not world_size//8 (#3); apply Buffer.set_num_sms so num_comm_sms is real (#8); nccl-tests -c 1 with a missing check footer is now invalid (#7); use context managers for file reads (#4,#5); launchers export COLLECTIVEX_IMAGE/_DIGEST for provenance (#9); trim workflow_dispatch sku options to launcher-backed pools (#2). Artifact-path finding (#6) already fixed via cx_collect_results.
Oseltamivir added a commit that referenced this pull request Aug 7, 2026
@Oseltamivir
CollectiveX: delete MoRI's unreachable registered-input path and the …
a45643e 
…spent DeepEP rewrite

Two dead paths, from an external review of the PR (queries.md items #4 and #5).

MoRI: `_external_input` is assigned once, True, and read nowhere else in the tree -- no env var,
no config key, no test -- so the registered-buffer arms of stage() and combine_transformed(), and
the `or not self._external_input` in stage_device_work, cannot execute. Removed, keeping the
external-input implementation and the note that 16 combine warps belong to this mode (MoRI's
tuned tables key combine on zero_copy, so the warp count and the buffer mode have to move
together).

Also fixes a latent defect the review did not name: expected_config compared the realized
inter-node config against a hardcoded "gpu_per_node": 8. Not live -- mori is registered at EP8
only, on three SKUs that all have 8 GPUs per node -- but it defeats the check's purpose, which is
to prove requested == realized. The moment mori scale-out ran on a non-8-GPU node it would abort
with a spurious "configuration was not realized". Now reads gpus_per_node. The neighbouring
rdma_block_num/num_qp_per_pe literals stay: those are genuinely file-invariant.

DeepEP: the pin tracks upstream main, whose deep_ep/__init__.py already carries the 'libnccl'
scan narrowing (#640), so rewrite_deepep_v2 took its early return on every run. Deleting it
rather than keeping a harmless no-op, because it is not harmless: the guard accepts exactly two
literal string forms, so any future upstream edit to that line -- a rename, requote or line split
-- falls through to `count(old) != 1` and exits 1, aborting every deepep-v2 leg at source-prep.
It converts routine upstream drift into a fleet-wide tripwire while guarding nothing. Removed
with its SPECS entry and common.sh invocation together, so the stage.py/common.sh lockstep holds
(six subcommands declared, six invoked, none orphaned).

The review also listed "the tests proving the no-op succeeds" for deletion; there are none.

Tests unchanged, 150 green. The MoRI file executes only on the AMD benchmark images, so the
deletion is behaviour-preserving by construction rather than by test -- a routine mori leg is the
on-metal confirmation.
xiaohuguo2023 added a commit that referenced this pull request Aug 19, 2026
@xiaohuguo2023 @claude
[k3-fp4][WIP] bump base image to vLLM ROCm nightly-8efa13b70 (0.27.2)
f35f8f8 
Move the K3 FP4 MI355X recipe onto vllm/vllm-openai-rocm:nightly-8efa13b700f1836657699cae2503dc2feab27fa0
(vLLM 0.27.2rc1.dev122+g8efa13b70, ROCm 7.2.3): 3 image: pins in
configs/amd-master.yaml (:627/:1296/:1325) + doc/comment tag copies +
docs/waiver/2508.md. Also carries the native) KV-offload arm in the base
entry script.

WIP — NOT yet validated on the new image: 0.27 rewrote rocm_aiter_mla.py,
so the patch orchestrator dies at 5/9 (patch_fp8asm.py padded_q anchor).
Patch reconciliation (drop upstreamed patches #1/#5/#6 + trio #2/#3/#4,
keep #7 offload eagle veto, re-port dspark 4/7,4c,5/7) + conc-1 DSpark
boot/accuracy + perf compare pending. Do not merge until green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@chunfangamd chunfangamd mentioned this pull request Aug 20, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kimbochen kimbochen kimbochen left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kedarpotdar-nv @kimbochen