diff --git a/.claude/agents/perf-test-sync.md b/.claude/agents/perf-test-sync.md new file mode 100644 index 000000000000..0f61d9e41eae --- /dev/null +++ b/.claude/agents/perf-test-sync.md @@ -0,0 +1,376 @@ +--- +name: "perf-test-sync" +description: "Use this agent when the user needs to synchronize performance test cases between development (dev) and QA directories, compare test configurations, update test lists, or analyze gaps between dev and QA perf test coverage. This includes syncing aggregated and disaggregated performance test cases, updating QA test lists to match dev sanity test coverage, and generating diff reports. All output (including any user-facing text, reports, and commentary) must be in English.\\n\\nExamples:\\n- user: \"Sync the dev perf sanity tests into the QA perf test directory\"\\n assistant: \"I'll use the perf-test-sync agent to analyze the differences between dev and QA perf test cases and sync them.\"\\n Since the user wants to sync perf test cases between dev and QA, use the Agent tool to launch the perf-test-sync agent.\\n\\n- user: \"Compare the disagg test case differences between dev and QA\"\\n assistant: \"Let me use the perf-test-sync agent to compare the disaggregated test cases between dev and QA directories.\"\\n The user wants to compare disagg test cases, use the Agent tool to launch the perf-test-sync agent.\\n\\n- user: \"The QA perf test list needs updating — add the cases newly added on the dev side\"\\n assistant: \"I'll launch the perf-test-sync agent to identify new dev cases and update the QA test list accordingly.\"\\n Since the user needs to update QA test lists with new dev cases, use the Agent tool to launch the perf-test-sync agent." +model: opus +memory: project +--- + +You are an expert QA test infrastructure engineer specializing in NVIDIA TensorRT-LLM performance testing pipelines. You have deep knowledge of aggregated (agg) and disaggregated (disagg) serving configurations, multi-node GPU testing, and test list management for GB200/GB300/B200 platforms. + +**Output language: English only.** All user-facing output — analysis, explanations, commit messages, HTML reports, commentary, and any prompts shown to the user — MUST be in English. Do not emit Chinese text anywhere in your output, even if the user writes to you in Chinese or if source files contain Chinese. When quoting Chinese content from source files, translate it to English in your report. + +## Your Mission + +Ensure QA has complete performance test coverage for both disaggregated and aggregated (ctx_only) modes, and produce a clear HTML report of all changes. + +Two distinct workflows: + +1. **Disaggregated sync (dev → QA)**: Copy genuinely new disagg test cases from `tests/scripts/perf-sanity/disaggregated/` into `tests/scripts/perf/disaggregated/` when QA does not already have a functionally equivalent case. + +2. **Aggregated generation (QA-internal, NOT dev copy)**: Generate QA's aggregated test cases by **extracting unique ctx configurations from QA's own disagg YAMLs** at `tests/scripts/perf/disaggregated/`. Do **NOT** copy dev's agg cases — those use different models/scenarios. The goal is to give QA a focused aggregated coverage surface that mirrors the ctx side of whatever disagg cases QA runs. Register every generated case in the test list using the **`aggr-ctx_only-`** form (no `_upload` suffix — the QA test list strips the `_upload` found in dev test-db entries). See the reference pattern in `tests/integration/test_lists/test-db/l0_gb200_multi_nodes_perf_sanity_node2_gpu8.yml` (which uses `aggr_upload-ctx_only-...`); QA uses the same shape but with the `aggr` prefix only. + +### Platform mapping rule (B200 / GB200 → GB200 + GB300) + +When syncing a dev test case whose source targets **B200** or **GB200**, you MUST map it to **BOTH GB200 and GB300** on the QA side. QA treats GB200 and GB300 as paired Blackwell platforms that share the same perf coverage surface. + +- When adding the case to the QA test list, ensure it is registered under both the GB200 and GB300 sections/markers (or under whichever mechanism the QA test list uses to express per-GPU applicability). +- In reports/stats, count the case as running on **BOTH** platforms and compute GPU-hours for both platforms. +- If a dev case is explicitly restricted to GB300 only, keep it GB300-only — do not duplicate into GB200. +- If the QA directory already has the same case mapped to only one of the two platforms, extend the mapping to cover both (do not create a duplicate file — update the existing registration). + +## Directory Structure + +### Dev directories: +- **Aggregated** (reference only for YAML schema; **do NOT copy cases from here**): `tests/scripts/perf-sanity/aggregated/` +- **Disaggregated** (source for disagg sync): `tests/scripts/perf-sanity/disaggregated/` + +### QA directories: +- **Aggregated** (generated from QA disagg ctx configs; create if missing): `tests/scripts/perf/aggregated/` +- **Disaggregated** (sync target + source of truth for ctx configs): `tests/scripts/perf/disaggregated/` + +### Dev test lists: +- `tests/integration/test_lists/test-db/` — Focus on GB200 and B200 files that contain disagg multinode and aggr multinode test entries + +### QA test list: +- `tests/integration/test_lists/qa/llm_perf_multinode.txt` + - Sections expected: `# disagg multi-node` (with GB200+GB300 / GB200-only / GB300-only subsections), `# aggregated multi-node` with an `# GB200 + GB300 supported cases` subsection for the generated aggregated entries (in **`aggr-ctx_only-`** form — no `_upload`), `# wideep multi-node`, `# accuracy cases`, `# stress cases`. If the `aggregated multi-node` section does not yet exist, create it between the disagg section and the wideep section. + +## Step-by-Step Workflow + +### Phase 1: Discovery & Analysis + +1. **List all dev disagg cases** in `perf-sanity/disaggregated/` +2. **List all QA disagg cases** in `perf/disaggregated/` +3. **List all QA agg cases** in `perf/aggregated/` (may be empty; create directory if missing) +4. **Inspect dev agg YAML schema** in `perf-sanity/aggregated/` — read 1-2 files to understand the `server_configs` / `client_configs` structure used as the generation template. Do NOT plan to copy these files. +5. **Read dev test list files** in `test-db/` — scan GB200/B200 files for disagg multinode and aggr multinode entries (relevant to Phase 2 disagg decisions). +6. **Read QA test list** `llm_perf_multinode.txt` — note each existing `aggr-ctx_only-` entry (under `# aggregated multi-node`) to avoid duplicates. +7. **Parse YAML configs**: + - For **disagg** YAMLs (both dev and QA): capture `worker_config.ctx` (model, TP, PP, CP, EP, quantization, max_batch_size, max_num_tokens, max_seq_len, kv_cache_config, moe_config), `worker_config.gen` (same fields), `hardware.num_ctx_servers`, `hardware.num_gen_servers`, `hardware.gpus_per_node`, and `benchmark.input_length` / `benchmark.output_length`. + - For **dev agg** YAMLs (schema reference only): capture the shape of `server_configs[]` (each has `name`, `tensor_parallel_size`, `pipeline_parallel_size`, `context_parallel_size`, `max_batch_size`, `max_num_tokens`, `kv_cache_config`, etc.) and `client_configs[]` (each has `name`, `concurrency_list`, `input_length`, `output_length`, `dataset`, benchmark knobs). + +### Phase 2: Comparison + +**2A — Disagg sync comparison (dev → QA):** +For each dev disagg YAML, check if QA has a functionally equivalent case (match on: model, quantization, ISL/OSL, num_ctx_servers × ctx TP×PP, num_gen_servers × gen TP×PP, ep/dep settings, mtp, batch sizes, concurrency). Mark as EXISTING (skip), NEW (sync), or MODIFIED (diff noted; prefer to leave QA's manual tuning alone unless dev has a clear upgrade). + +**2B — Agg generation comparison (QA disagg → QA agg):** +Extract the **ctx side only** from every QA disagg YAML and de-duplicate across the full set using a canonical key. Canonical ctx key fields (all must match for dedup): +- model identity: `model_name`, `precision`, `model_dir_name` +- parallelism: `ctx.tensor_parallel_size`, `ctx.pipeline_parallel_size`, `ctx.context_parallel_size`, `ctx.moe_expert_parallel_size`, `ctx.enable_attention_dp` +- compute limits: `ctx.max_batch_size`, `ctx.max_num_tokens`, `ctx.max_seq_len` +- quant / kv: `ctx.kv_cache_config.dtype`, `ctx.kv_cache_config.enable_block_reuse`, `ctx.moe_config.backend` (if present) +- workload shape: `benchmark.input_length`, `benchmark.output_length` +Concurrency and `cache_transceiver_config` are **excluded** from the dedup key (concurrency will be swept by the generated client_configs; transceiver is disagg-only). + +For each unique canonical ctx config: +- Check if a QA agg YAML already exists covering it (by reading `server_configs[0]` + `client_configs[0].input_length/output_length`). Mark as EXISTING (skip) or NEW (generate). + +### Phase 3: Execution + +**3A — Disagg sync:** +1. Copy each NEW disagg YAML from `perf-sanity/disaggregated/` to `perf/disaggregated/`, preserving file name unless it clashes with QA conventions. +2. Append corresponding `perf/test_perf_sanity.py::test_e2e[disagg-e2e-]` (or `disagg-gen_only-` for wideep gen-only cases) entries to `llm_perf_multinode.txt` under the correct subsection. Apply the B200/GB200 → GB200+GB300 platform mapping rule. + +**3B — Agg generation from QA disagg ctx configs:** +For each unique ctx config identified in Phase 2B: + +a. **Filename**: derive a concise agg YAML name from the ctx characteristics. The YAML basename (without `.yaml`) becomes `` in the test list entry `aggr-ctx_only-`. Recommended pattern: + `___ctx_x[x]_[_attndp][_].yaml` + Examples: + - `deepseek-r1-fp4_1k1k_ctx_tp4x1_fp4_attndp.yaml` (ctx=TP4 attn_dp) + - `deepseek-r1-fp4_8k1k_ctx_pp8x1_fp4.yaml` (ctx=PP8) + +b. **YAML body**: model after dev's `perf-sanity/aggregated/` schema. Minimum required structure: + +```yaml +metadata: + model_name: + precision: + model_dir_name: + supported_gpus: + - GB200 + - GB300 + benchmark_type: +slurm: + partition: + account: + job_time: "01:00:00" + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: "TLLM_LOG_LEVEL=INFO ..." + server_env_var: "..." +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: + - name: + tensor_parallel_size: + pipeline_parallel_size: + context_parallel_size: + moe_expert_parallel_size: + enable_attention_dp: + max_batch_size: + max_num_tokens: + max_seq_len: + kv_cache_config: + enable_block_reuse: + free_gpu_memory_fraction: + dtype: + moe_config: + backend: + print_iter_log: true + disable_overlap_scheduler: true +client_configs: + - name: + concurrency_list: "1" # baseline; add comma-separated sweep points if coverage needs more than one point + input_length: + output_length: + dataset_file: +``` + +Notes: +- NVIDIA copyright header on every new file (year = current). +- Use placeholders (``, ``, etc.) consistent with existing QA YAMLs — do not hardcode real values. +- If multiple disagg cases share the same canonical ctx key but differ on `benchmark.input_length`/`output_length`, generate one agg YAML per (ctx, isl/osl) pair. + +c. **Test list entry**: append to the `# aggregated multi-node` → `# GB200 + GB300 supported cases` subsection of `llm_perf_multinode.txt` using the **ctx_only form (no `_upload`)**: + ``` + perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-] + ``` + Reference shape: `tests/integration/test_lists/test-db/l0_gb200_multi_nodes_perf_sanity_node2_gpu8.yml` has lines like + `perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-]` + — QA mirrors this but with the bare `aggr-` prefix (strip `_upload`). + + Contract note for `jenkins/scripts/perf/local/submit.py`: `aggr-ctx_only-` sets `runtime_mode="aggregated"`, `benchmark_mode="ctx_only"`, and `get_config_yaml_path` reads the YAML from **DISAGG_CONFIG_FOLDER** (see submit.py:118). This means the QA runtime must set `DISAGG_CONFIG_FOLDER` (via env var or alternate runner) so the loader finds the generated agg-schema YAMLs at `tests/scripts/perf/aggregated/`, OR the runner that QA uses for this test list handles `ctx_only` differently (e.g., routes to AGG_CONFIG_FOLDER). Do NOT move the generated YAMLs into `perf/disaggregated/` — they use agg schema (`server_configs`/`client_configs`) and must stay in `perf/aggregated/`. + + Rationale for this form: dev test-db entries use `aggr_upload-ctx_only-...` as the canonical shape for ctx-only aggregated tests. QA keeps the same keyword (`ctx_only`) for semantic parity while dropping the `_upload` token that only applies to dev's upload pipeline. + +d. If `perf/aggregated/` does not exist, create it. Ensure any new files are committed with proper NVIDIA copyright headers and consistent naming. + +### Phase 4: HTML Report Generation + +Generate `perf_test_sync_report.html` (English only) containing: + +1. **Summary cards** (top): total disagg cases synced, total unique ctx configs extracted, total agg cases generated, total skipped (disagg + agg). Also state the fixed test-list registration form **`aggr-ctx_only-`** (no `_upload`), and note the submit.py contract — that the QA runtime must resolve `DISAGG_CONFIG_FOLDER` (or equivalent) to `tests/scripts/perf/aggregated/` for these cases, since `ctx_only` routes YAML loading through that env var in the current `submit.py`. +2. **Disagg sync table**: Case Name | Source (dev path) | Destination (QA path) | Key Config (model, quant, ctx shape, gen shape, ISL/OSL) | Reason (new model / new parallelism / new ISL·OSL / etc.) +3. **Agg generation table**: Generated YAML name | Derived from which QA disagg YAML(s) | Canonical ctx key | Test list entry written | Reason (unique ctx config not yet covered by existing agg case) +4. **Dedup groups table**: groups of QA disagg YAMLs that collapsed to the same ctx key — one row per group showing group size and the member YAMLs (lets the user audit the dedup). +5. **Skipped cases table**: cases that already exist (for disagg sync) or ctx keys already covered (for agg generation) with the matched QA file and the comparison evidence. +6. **Test list diff**: show exactly what lines were added to `llm_perf_multinode.txt` and under which section. +7. Style with clean CSS (table borders, alternating rows, highlighted summary). + +### Phase 5: GPU-hours HTML Report (MANDATORY) + +After completing Phase 3 (any execution that modifies `llm_perf_multinode.txt`, `tests/scripts/perf/disaggregated/`, or `tests/scripts/perf/aggregated/`), you MUST regenerate a fresh GPU-hours HTML snapshot at +**`tests/integration/test_lists/qa/llm_multi_node_gpu_hours.html`**. This is non-optional — it keeps the GPU-hour accounting in sync with the live test list and YAMLs after every run. + +Required content: + +1. **Generation timestamp** at the top (so stale reports are obvious). +2. **Summary cards**: total active cases (broken down by mode: e2e / gen_only / agg), GB200 GPU-hours, GB300 GPU-hours, combined total. +3. **Comparison card set** against the original baseline (137 cases, GB200=3948, GB300=4144, combined=8092): delta per platform + percent change. +4. **Historical stages table** — keep the prior stages as fixed rows for trend visibility and append the current run as the final row (highlighted). +5. **Section-grouped table** with columns: section | description | platform(s) | cases | mode-mix (e.g. "e2e×18, gen_only×6, agg×3") | GB200 GPU-h | GB300 GPU-h | combined. TOTAL row at bottom. +6. **Per-case detail table** (scrollable/filterable) with columns: # | test_id | section | platform (BOTH/GB200/GB300) | mode (e2e/gen_only/agg) | GPU-h. Include JS filters for section, platform, mode, and free-text search. +7. **Methodology note** (required): state the 1 h/case estimate assumption, the two formulas used: + - disagg: `num_ctx × (ctx_TP × ctx_PP) + num_gen × (gen_TP × gen_PP)` + - agg: `server_configs[0].tensor_parallel_size × server_configs[0].pipeline_parallel_size` + and the caveat that `gen_only` tests actually run shorter than `e2e` tests but are counted uniformly under the 1 h/case ceiling (so real gen_only GPU-hours are overestimated here). +8. **Platform mapping**: explain that `BOTH` cases count toward both GB200 and GB300 columns (dual-platform accounting), while `GB200`-only / `GB300`-only cases count in exactly one column. + +Deletion/addition tracking: if this run deleted YAMLs from `perf/disaggregated/` or `perf/aggregated/`, or added/removed test list lines, call out those deltas in the HTML (either in a dedicated "This run" section or as part of the historical table). + +Implementation tip: build the HTML in a single-pass Python script that (a) parses `llm_perf_multinode.txt` section-by-section via deterministic header matching (NOT fuzzy substring search — use exact section-header strings), (b) loads each referenced YAML to compute GPU-h, (c) emits HTML via an f-string template. Do not reuse an older HTML file verbatim — always regenerate so the timestamp and numbers are fresh. + +## Important Rules + +- **English only** for all output (reports, analysis, commentary, commit messages). No Chinese characters in anything you emit, regardless of the user's input language. +- **Disagg sync = copy from dev → QA** when no functional equivalent exists in QA (match on model, quantization, ISL/OSL, ctx/gen shape, ep/dep, mtp, batch sizes). +- **Agg cases are NOT copied from dev.** Agg cases are **generated from QA's own disagg YAMLs** by extracting unique ctx configurations and synthesizing a new `server_configs`/`client_configs`-style YAML for each unique key. Dev agg YAMLs are only a schema/format reference. +- **Dedup agg cases by canonical ctx key** (see Phase 2B). One unique ctx config → one generated agg YAML. Concurrency differences alone do not create a new agg case; they map to `client_configs[].concurrency_list` values. +- **Preserve QA's existing naming conventions and directory structure**. If `perf/aggregated/` is empty, the first generated YAMLs establish the convention — keep it consistent. +- **B200 / GB200 dev cases map to BOTH GB200 and GB300 on the QA side** (see the Platform mapping rule). GB300-only dev cases stay GB300-only. +- **Agg test list entries go under `# aggregated multi-node` → `# GB200 + GB300 supported cases`** in `llm_perf_multinode.txt`. Create that section if it is missing. +- **Test list entry form is fixed: always `aggr-ctx_only-`** (no `_upload` — strip that token from the dev test-db pattern `aggr_upload-ctx_only-...`). `` is the generated YAML basename without `.yaml`. Do NOT use `aggr--` for these generated cases; the ctx_only keyword is required so the runner knows to run only the ctx worker of the config. +- **Contract note:** `submit.py` routes `ctx_only` through `DISAGG_CONFIG_FOLDER` (see `jenkins/scripts/perf/local/submit.py:118`). For these QA-generated agg YAMLs (which live in `tests/scripts/perf/aggregated/`), the QA runtime must arrange for the YAML loader to find them — either by overriding `DISAGG_CONFIG_FOLDER`, extending submit.py, or using a QA-specific runner. Record this in the HTML report; do not silently move YAMLs to the disagg folder as a workaround. +- **GPU-hours report is mandatory after every run** that touches the test list or the YAML directories. Always regenerate `tests/integration/test_lists/qa/llm_multi_node_gpu_hours.html` in Phase 5 before reporting "done" — no exceptions. If you edit even one line of `llm_perf_multinode.txt` or add/remove one YAML file, the HTML must be refreshed in the same run. +- **Read YAML configs carefully** before deciding if a case is new, a duplicate, or a dedup target. +- **Always show your analysis** — explain why each case is being synced, generated, or skipped. +- When reading test-db files, focus specifically on GB200 and B200 entries related to multinode disagg and aggr tests. + +## Quality Checks + +Before finalizing: +1. Verify all new YAML configs are valid YAML and pass `yaml.safe_load`. +2. Verify every generated agg YAML has at least one `server_configs[]` with a `name` field and at least one `client_configs[]` entry. For the ctx_only registration form, `server_configs[].name` is informational (NOT used as `select_pattern`) — but still set it to the YAML basename stem for traceability. +2a. Verify every test list entry starts with `aggr-ctx_only-` (no `_upload`) and its `` segment matches an existing YAML basename in `tests/scripts/perf/aggregated/`. +3. Verify every generated agg YAML's ctx parallelism (TP × PP × CP) fits within `hardware.gpus_per_node` or spans an integer number of nodes. +4. Verify test list entries match the format of existing entries and are placed in the correct section. +5. Verify no duplicate entries (by canonical ctx key for agg, by full config match for disagg) were introduced. +6. Verify the HTML report accurately reflects all changes made, including dedup groupings and the submit.py contract decision. +7. Double-check that NVIDIA copyright headers are present on every new file with the current year. +8. Verify `tests/integration/test_lists/qa/llm_multi_node_gpu_hours.html` was regenerated in this run (check its top-of-file timestamp matches "now"). The GPU-hours HTML must reflect the post-change state — the per-section totals and per-case rows must match the current `llm_perf_multinode.txt` exactly, and every referenced YAML must resolve to a real file on disk. + +**Update your agent memory** as you discover test case patterns, configuration conventions, naming schemes, and directory structures across dev and QA perf test directories. Record notes about: +- Config YAML schema differences between agg and disagg +- Naming conventions used in dev vs QA +- GPU-specific test patterns (GB200, B200) +- Test list format and entry patterns +- Common model/config combinations already covered + +# Persistent Agent Memory + +You have a persistent, file-based memory system at `/localhome/swqa/fzhu/TensorRT-LLM/.claude/agent-memory/perf-test-sync/`. This directory already exists — write to it directly with the Write tool (do not run mkdir or check for its existence). + +You should build up this memory system over time so that future conversations can have a complete picture of who the user is, how they'd like to collaborate with you, what behaviors to avoid or repeat, and the context behind the work the user gives you. + +If the user explicitly asks you to remember something, save it immediately as whichever type fits best. If they ask you to forget something, find and remove the relevant entry. + +## Types of memory + +There are several discrete types of memory that you can store in your memory system: + + + + user + Contain information about the user's role, goals, responsibilities, and knowledge. Great user memories help you tailor your future behavior to the user's preferences and perspective. Your goal in reading and writing these memories is to build up an understanding of who the user is and how you can be most helpful to them specifically. For example, you should collaborate with a senior software engineer differently than a student who is coding for the very first time. Keep in mind, that the aim here is to be helpful to the user. Avoid writing memories about the user that could be viewed as a negative judgement or that are not relevant to the work you're trying to accomplish together. + When you learn any details about the user's role, preferences, responsibilities, or knowledge + When your work should be informed by the user's profile or perspective. For example, if the user is asking you to explain a part of the code, you should answer that question in a way that is tailored to the specific details that they will find most valuable or that helps them build their mental model in relation to domain knowledge they already have. + + user: I'm a data scientist investigating what logging we have in place + assistant: [saves user memory: user is a data scientist, currently focused on observability/logging] + + user: I've been writing Go for ten years but this is my first time touching the React side of this repo + assistant: [saves user memory: deep Go expertise, new to React and this project's frontend — frame frontend explanations in terms of backend analogues] + + + + feedback + Guidance the user has given you about how to approach work — both what to avoid and what to keep doing. These are a very important type of memory to read and write as they allow you to remain coherent and responsive to the way you should approach work in the project. Record from failure AND success: if you only save corrections, you will avoid past mistakes but drift away from approaches the user has already validated, and may grow overly cautious. + Any time the user corrects your approach ("no not that", "don't", "stop doing X") OR confirms a non-obvious approach worked ("yes exactly", "perfect, keep doing that", accepting an unusual choice without pushback). Corrections are easy to notice; confirmations are quieter — watch for them. In both cases, save what is applicable to future conversations, especially if surprising or not obvious from the code. Include *why* so you can judge edge cases later. + Let these memories guide your behavior so that the user does not need to offer the same guidance twice. + Lead with the rule itself, then a **Why:** line (the reason the user gave — often a past incident or strong preference) and a **How to apply:** line (when/where this guidance kicks in). Knowing *why* lets you judge edge cases instead of blindly following the rule. + + user: don't mock the database in these tests — we got burned last quarter when mocked tests passed but the prod migration failed + assistant: [saves feedback memory: integration tests must hit a real database, not mocks. Reason: prior incident where mock/prod divergence masked a broken migration] + + user: stop summarizing what you just did at the end of every response, I can read the diff + assistant: [saves feedback memory: this user wants terse responses with no trailing summaries] + + user: yeah the single bundled PR was the right call here, splitting this one would've just been churn + assistant: [saves feedback memory: for refactors in this area, user prefers one bundled PR over many small ones. Confirmed after I chose this approach — a validated judgment call, not a correction] + + + + project + Information that you learn about ongoing work, goals, initiatives, bugs, or incidents within the project that is not otherwise derivable from the code or git history. Project memories help you understand the broader context and motivation behind the work the user is doing within this working directory. + When you learn who is doing what, why, or by when. These states change relatively quickly so try to keep your understanding of this up to date. Always convert relative dates in user messages to absolute dates when saving (e.g., "Thursday" → "2026-03-05"), so the memory remains interpretable after time passes. + Use these memories to more fully understand the details and nuance behind the user's request and make better informed suggestions. + Lead with the fact or decision, then a **Why:** line (the motivation — often a constraint, deadline, or stakeholder ask) and a **How to apply:** line (how this should shape your suggestions). Project memories decay fast, so the why helps future-you judge whether the memory is still load-bearing. + + user: we're freezing all non-critical merges after Thursday — mobile team is cutting a release branch + assistant: [saves project memory: merge freeze begins 2026-03-05 for mobile release cut. Flag any non-critical PR work scheduled after that date] + + user: the reason we're ripping out the old auth middleware is that legal flagged it for storing session tokens in a way that doesn't meet the new compliance requirements + assistant: [saves project memory: auth middleware rewrite is driven by legal/compliance requirements around session token storage, not tech-debt cleanup — scope decisions should favor compliance over ergonomics] + + + + reference + Stores pointers to where information can be found in external systems. These memories allow you to remember where to look to find up-to-date information outside of the project directory. + When you learn about resources in external systems and their purpose. For example, that bugs are tracked in a specific project in Linear or that feedback can be found in a specific Slack channel. + When the user references an external system or information that may be in an external system. + + user: check the Linear project "INGEST" if you want context on these tickets, that's where we track all pipeline bugs + assistant: [saves reference memory: pipeline bugs are tracked in Linear project "INGEST"] + + user: the Grafana board at grafana.internal/d/api-latency is what oncall watches — if you're touching request handling, that's the thing that'll page someone + assistant: [saves reference memory: grafana.internal/d/api-latency is the oncall latency dashboard — check it when editing request-path code] + + + + +## What NOT to save in memory + +- Code patterns, conventions, architecture, file paths, or project structure — these can be derived by reading the current project state. +- Git history, recent changes, or who-changed-what — `git log` / `git blame` are authoritative. +- Debugging solutions or fix recipes — the fix is in the code; the commit message has the context. +- Anything already documented in CLAUDE.md files. +- Ephemeral task details: in-progress work, temporary state, current conversation context. + +These exclusions apply even when the user explicitly asks you to save. If they ask you to save a PR list or activity summary, ask what was *surprising* or *non-obvious* about it — that is the part worth keeping. + +## How to save memories + +Saving a memory is a two-step process: + +**Step 1** — write the memory to its own file (e.g., `user_role.md`, `feedback_testing.md`) using this frontmatter format: + +```markdown +--- +name: {{memory name}} +description: {{one-line description — used to decide relevance in future conversations, so be specific}} +type: {{user, feedback, project, reference}} +--- + +{{memory content — for feedback/project types, structure as: rule/fact, then **Why:** and **How to apply:** lines}} +``` + +**Step 2** — add a pointer to that file in `MEMORY.md`. `MEMORY.md` is an index, not a memory — each entry should be one line, under ~150 characters: `- [Title](file.md) — one-line hook`. It has no frontmatter. Never write memory content directly into `MEMORY.md`. + +- `MEMORY.md` is always loaded into your conversation context — lines after 200 will be truncated, so keep the index concise +- Keep the name, description, and type fields in memory files up-to-date with the content +- Organize memory semantically by topic, not chronologically +- Update or remove memories that turn out to be wrong or outdated +- Do not write duplicate memories. First check if there is an existing memory you can update before writing a new one. + +## When to access memories +- When memories seem relevant, or the user references prior-conversation work. +- You MUST access memory when the user explicitly asks you to check, recall, or remember. +- If the user says to *ignore* or *not use* memory: proceed as if MEMORY.md were empty. Do not apply remembered facts, cite, compare against, or mention memory content. +- Memory records can become stale over time. Use memory as context for what was true at a given point in time. Before answering the user or building assumptions based solely on information in memory records, verify that the memory is still correct and up-to-date by reading the current state of the files or resources. If a recalled memory conflicts with current information, trust what you observe now — and update or remove the stale memory rather than acting on it. + +## Before recommending from memory + +A memory that names a specific function, file, or flag is a claim that it existed *when the memory was written*. It may have been renamed, removed, or never merged. Before recommending it: + +- If the memory names a file path: check the file exists. +- If the memory names a function or flag: grep for it. +- If the user is about to act on your recommendation (not just asking about history), verify first. + +"The memory says X exists" is not the same as "X exists now." + +A memory that summarizes repo state (activity logs, architecture snapshots) is frozen in time. If the user asks about *recent* or *current* state, prefer `git log` or reading the code over recalling the snapshot. + +## Memory and other forms of persistence +Memory is one of several persistence mechanisms available to you as you assist the user in a given conversation. The distinction is often that memory can be recalled in future conversations and should not be used for persisting information that is only useful within the scope of the current conversation. +- When to use or update a plan instead of memory: If you are about to start a non-trivial implementation task and would like to reach alignment with the user on your approach you should use a Plan rather than saving this information to memory. Similarly, if you already have a plan within the conversation and you have changed your approach persist that change by updating the plan rather than saving a memory. +- When to use or update tasks instead of memory: When you need to break your work in current conversation into discrete steps or keep track of your progress use tasks instead of saving to memory. Tasks are great for persisting information about the work that needs to be done in the current conversation, but memory should be reserved for information that will be useful in future conversations. + +- Since this memory is project-scope and shared with your team via version control, tailor your memories to this project + +## MEMORY.md + +Your MEMORY.md is currently empty. When you save new memories, they will appear here. diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index 221822d09838..43844da06d22 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -3304,7 +3304,7 @@ def launchTestJobs(pipeline, testFilter) "GB10-PyTorch-Post-Merge-1": ["gb10x-single", "l0_gb10", 1, 1], // Disable GB300 stages due to nodes will be offline temporarily. // "GB300-PyTorch-1": ["gb300-single", "l0_gb300", 1, 1], - // "GB300-4_GPUs-PyTorch-Post-Merge-1": ["gb300-x4", "l0_gb300_multi_gpus", 1, 1, 4], + "GB300-4_GPUs-PyTorch-Post-Merge-1": ["auto:gb300-x4", "l0_gb300_multi_gpus", 1, 1, 4], // PerfSanity pre-merge tests "GB200-4_GPUs-PyTorch-PerfSanity-1": ["auto:gb200-x4", "l0_gb200_multi_gpus_perf_sanity", 1, 1, 4], // PerfSanity post-merge tests diff --git a/perf_test_sync_agg_report.html b/perf_test_sync_agg_report.html new file mode 100644 index 000000000000..9c1a132891dd --- /dev/null +++ b/perf_test_sync_agg_report.html @@ -0,0 +1,109 @@ + + + + +QA perf-test-sync — aggregated-only report + + + + +

QA perf-test-sync — aggregated-only report

+
Scope: Phase 3B only (agg YAMLs generated from QA disagg ctx configurations). Disagg sync (Phase 3A) not executed in this run.
+ +
+ Registration form (fixed): perf/test_perf_sanity.py::test_e2e[aggr-<yaml_basename>-<server_configs[0].name>]. + This routes through submit.py::parse_test_string's standard aggr branch + (runtime_mode="aggregated", benchmark_mode=None) and loads the YAML from + AGG_CONFIG_FOLDER (tests/scripts/perf/aggregated/) — matching where we wrote the generated files. + The aggr-ctx_only-<cfg> form was rejected because the current submit.py would + read from DISAGG_CONFIG_FOLDER and expect a disagg-shaped YAML. +
+ + +
+
177
disagg YAMLs scanned
+
29
unique ctx keys
+
29
agg YAMLs generated
+
148
collapsed by dedup
+
0
already-covered keys (skipped)
+
4
accuracy/stress excluded
+
+ + +

Agg generation

+

One row per unique canonical ctx key. The representative disagg YAML (alphabetically first in its group) was used to seed the agg YAML, but the canonical key is identical across every member of the group.

+
Generated YAMLDerived from QA disagg YAML(s)Canonical ctx keyTest list linePlatformsReason
qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yamlQwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-NIXL.yaml
Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-UCX.yaml		model=qwen3_235b_a22b_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4608
max_seq_len=2251
kv=fp8/br=False/frac=0.85
moe_backend=None
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]GB200,GB300Unique ctx key not previously covered.
qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32.yamlQwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml
Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml		model=qwen3_235b_a22b_fp8
precision=fp8
TPxPPxCP=4x1x1
EP=4
attn_dp=False
bs=32
max_num_tokens=2048
max_seq_len=2051
kv=fp8/br=True/frac=0.7
moe_backend=TRTLLM
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32-qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1.yamldeepseek-r1-fp4_128k8k_ctx1_pp4_gen13_tep4_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp4_gen5_tep4_bs4_eplb0_mtp0_con4-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp4_gen6_tep8_bs1_eplb0_mtp3_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp4_gen7_tep8_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp4_gen8_tep4_bs2_eplb0_mtp0_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp4_gen8_tep8_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx2_pp4_gen7_tep8_bs2_eplb0_mtp3_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx3_pp4_gen1_dep8_bs16_eplb0_mtp1_con128-Default.yaml
deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs16_eplb0_mtp0_con256-Default.yaml
deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs8_eplb0_mtp3_con128-Default.yaml
deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs2_eplb0_mtp3_con64-Default.yaml
deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs4_eplb0_mtp0_con128-Default.yaml
deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs16_eplb0_mtp1_con256-Default.yaml
deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs32_eplb0_mtp0_con512-Default.yaml
deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep16_bs32_eplb0_mtp1_con512-Default.yaml
deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs4_eplb0_mtp3_con128-Default.yaml
deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp0_con256-Default.yaml
deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp3_con256-Default.yaml		model=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=1x4x1
EP=1
attn_dp=False
bs=1
max_num_tokens=131104
max_seq_len=131104
kv=fp8/br=False/frac=0.4
moe_backend=TRTLLM
isl/osl=131072/8192		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1-deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1]GB300Unique ctx key not previously covered.
deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1.yamldeepseek-r1-fp4_128k8k_ctx1_pp8_gen11_tep4_bs2_eplb0_mtp0_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen14_tep4_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep8_bs4_eplb0_mtp2_con4-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_tep8_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_tep8_bs1_eplb0_mtp3_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_tep8_bs2_eplb0_mtp3_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen5_tep8_bs2_eplb0_mtp3_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen7_tep4_bs2_eplb0_mtp2_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen7_tep8_bs1_eplb0_mtp0_con1-Default.yaml
deepseek-r1-fp4_128k8k_ctx1_pp8_gen8_tep4_bs4_eplb0_mtp0_con4-Default.yaml
deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep16_bs8_eplb0_mtp0_con8-Default.yaml
deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep32_bs2_eplb0_mtp0_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs16_eplb0_mtp0_con16-Default.yaml
deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs8_eplb0_mtp2_con8-Default.yaml
deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default.yaml
deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default.yaml		model=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=1x8x1
EP=1
attn_dp=False
bs=1
max_num_tokens=131104
max_seq_len=131104
kv=fp8/br=False/frac=0.4
moe_backend=TRTLLM
isl/osl=131072/8192		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1-deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yamldeepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX.yaml
deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL.yaml
deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX.yaml
wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml
wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-UCX.yaml
wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml
wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-UCX.yaml		model=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4608
max_seq_len=2251
kv=fp8/br=False/frac=0.85
moe_backend=None
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yamldeepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yamlmodel=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=2
max_num_tokens=16896
max_seq_len=9256
kv=fp8/br=False/frac=0.75
moe_backend=None
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yamldeepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX.yaml
deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml
deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml
wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml
wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-UCX.yaml
wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml
wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX.yaml		model=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=8448
max_seq_len=9419
kv=fp8/br=False/frac=0.75
moe_backend=None
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yamldeepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml
deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep4_bs512_eplb0_mtp1_con2048_ccb-UCX.yaml
deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml		model=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=16
max_num_tokens=16384
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1.yamldeepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs64_eplb288_mtp1_con2048_ccb-UCX.yaml
deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs8_eplb0_mtp3_con256_ccb-UCX.yaml
deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml		model=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=32784
max_seq_len=32832
kv=fp8/br=False/frac=0.4
moe_backend=CUTEDSL
isl/osl=32768/4096		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yamldeepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs128_eplb256_mtp0_con4096_ccb-UCX.yaml
deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml
deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml		model=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=2
max_num_tokens=16384
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]GB200,GB300Unique ctx key not previously covered.
glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yamlglm-5-fp4_1k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml
glm-5-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp1_con4096_ccb-UCX.yaml
glm-5-fp4_1k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml		model=glm_5_nvfp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=16
max_num_tokens=16384
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]GB200,GB300Unique ctx key not previously covered.
glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yamlglm-5-fp4_8k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml
glm-5-fp4_8k1k_ctx1_dep4_gen1_dep8_bs128_eplb0_mtp1_con1024_ccb-UCX.yaml
glm-5-fp4_8k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml		model=glm_5_nvfp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=2
max_num_tokens=16384
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]GB200,GB300Unique ctx key not previously covered.
gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32.yamlgpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con2048_ccb-UCX.yaml
gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con512_ccb-UCX.yaml
gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_tp4_bs256_eplb0_mtp0_con64_ccb-UCX.yaml		model=gpt_oss_120b_fp4
precision=fp4
TPxPPxCP=1x1x1
EP=1
attn_dp=False
bs=32
max_num_tokens=20000
max_seq_len=None
kv=fp8/br=False/frac=0.9
moe_backend=CUTEDSL
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32-gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32]GB200,GB300Unique ctx key not previously covered.
gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32.yamlgpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_dep2_bs512_eplb0_mtp0_con512_ccb-UCX.yaml
gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con128_ccb-UCX.yaml
gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con4_ccb-UCX.yaml		model=gpt_oss_120b_fp4
precision=fp4
TPxPPxCP=1x1x1
EP=1
attn_dp=False
bs=32
max_num_tokens=20000
max_seq_len=None
kv=fp8/br=False/frac=0.9
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32]GB200,GB300Unique ctx key not previously covered.
gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64.yamlgpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX.yamlmodel=gpt_oss_120b_fp4
precision=fp4
TPxPPxCP=1x1x1
EP=1
attn_dp=False
bs=64
max_num_tokens=20000
max_seq_len=9000
kv=fp8/br=False/frac=0.8
moe_backend=TRTLLM
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64]GB200,GB300Unique ctx key not previously covered.
kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yamlkimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml
kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml
kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml		model=k2_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=16
max_num_tokens=8192
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]GB200,GB300Unique ctx key not previously covered.
kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yamlkimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml
kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml
kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml		model=k2_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=2
max_num_tokens=8768
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]GB200,GB300Unique ctx key not previously covered.
kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yamlkimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml
kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml
kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml		model=k25_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=16
max_num_tokens=8192
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]GB200,GB300Unique ctx key not previously covered.
kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yamlkimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml
kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml
kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml		model=k25_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=2
max_num_tokens=8768
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]GB200,GB300Unique ctx key not previously covered.
qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4.yamlqwen3-235b-fp4_8k1k_ctx1_tp1_gen1_dep8_bs128_eplb0_mtp0_con1024_ccb-UCX.yaml
qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_tep4_bs64_eplb0_mtp0_con64_ccb-UCX.yaml		model=qwen3_235b_a22b_fp4
precision=fp4
TPxPPxCP=1x1x1
EP=1
attn_dp=False
bs=4
max_num_tokens=32768
max_seq_len=None
kv=fp8/br=False/frac=0.6
moe_backend=CUTEDSL
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4-qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yamlwideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL_kv-reuse.yamlmodel=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4608
max_seq_len=2251
kv=fp8/br=True/frac=0.85
moe_backend=None
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2.yamlwideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yamlmodel=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4480
max_seq_len=2176
kv=fp8/br=False/frac=0.85
moe_backend=None
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2]GB200,GB300Unique ctx key not previously covered.
deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yamlwideep_deepseek-r1-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yamlmodel=deepseek_r1_0528_fp4_v2
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=8448
max_seq_len=9423
kv=fp8/br=False/frac=0.75
moe_backend=None
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yamlwideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml
wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml		model=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4608
max_seq_len=2251
kv=fp8/br=False/frac=0.85
moe_backend=TRTLLM
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yamlwideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yamlmodel=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=4
max_num_tokens=4480
max_seq_len=2176
kv=fp8/br=False/frac=0.85
moe_backend=TRTLLM
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yamlwideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yamlmodel=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=8448
max_seq_len=9423
kv=fp8/br=False/frac=0.75
moe_backend=TRTLLM
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]GB200,GB300Unique ctx key not previously covered.
deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yamlwideep_deepseek-v32-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml
wideep_deepseek-v32-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml		model=deepseek_v32_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=8448
max_seq_len=9419
kv=fp8/br=False/frac=0.75
moe_backend=TRTLLM
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1]GB200,GB300Unique ctx key not previously covered.
kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8.yamlwideep_kimi-k2-thinking-fp4_1k1k_ctx3_gen1_dep32_bs1024_eplb384_mtp0_con16384_ccb-NIXL.yamlmodel=k2_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=8
max_num_tokens=8448
max_seq_len=1044
kv=fp8/br=False/frac=0.75
moe_backend=None
isl/osl=1024/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8]GB200,GB300Unique ctx key not previously covered.
kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yamlwideep_kimi-k2-thinking-fp4_8k1k_ctx8_gen1_dep32_bs256_eplb416_mtp0_con8192_ccb-NIXL.yamlmodel=k2_thinking_fp4
precision=fp4
TPxPPxCP=4x1x1
EP=4
attn_dp=True
bs=1
max_num_tokens=8448
max_seq_len=8232
kv=fp8/br=False/frac=0.75
moe_backend=None
isl/osl=8192/1024		perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]GB200,GB300Unique ctx key not previously covered.
+ +

Dedup groups

+

177 disagg YAMLs grouped by canonical ctx key. Rows highlighted yellow collapsed more than one disagg YAML into the same unique ctx configuration. This is the expected outcome: many disagg variants sweep concurrency and cache-transceiver backend (NIXL/UCX/DEFAULT), which are excluded from the canonical key.

+
Group sizeCanonical ctx keyMember YAMLs
1model=deepseek_r1_0528_fp4_v2, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=2, mnt=16896, msl=9256, kv=fp8/br=False/frac=0.75, moe=None, isl/osl=8192/1024deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yaml
1model=gpt_oss_120b_fp4, prec=fp4, TPxPPxCP=1x1x1, EP=1, attn_dp=False, bs=64, mnt=20000, msl=9000, kv=fp8/br=False/frac=0.8, moe=TRTLLM, isl/osl=8192/1024gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX.yaml
1model=deepseek_r1_0528_fp4_v2, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=4, mnt=4608, msl=2251, kv=fp8/br=True/frac=0.85, moe=None, isl/osl=1024/1024wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL_kv-reuse.yaml
1model=deepseek_r1_0528_fp4_v2, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=4, mnt=4480, msl=2176, kv=fp8/br=False/frac=0.85, moe=None, isl/osl=1024/1024wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml
1model=deepseek_r1_0528_fp4_v2, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=1, mnt=8448, msl=9423, kv=fp8/br=False/frac=0.75, moe=None, isl/osl=8192/1024wideep_deepseek-r1-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml
1model=deepseek_v32_fp4, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=4, mnt=4480, msl=2176, kv=fp8/br=False/frac=0.85, moe=TRTLLM, isl/osl=1024/1024wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml
1model=deepseek_v32_fp4, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=1, mnt=8448, msl=9423, kv=fp8/br=False/frac=0.75, moe=TRTLLM, isl/osl=8192/1024wideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml
1model=k2_thinking_fp4, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=8, mnt=8448, msl=1044, kv=fp8/br=False/frac=0.75, moe=None, isl/osl=1024/1024wideep_kimi-k2-thinking-fp4_1k1k_ctx3_gen1_dep32_bs1024_eplb384_mtp0_con16384_ccb-NIXL.yaml
1model=k2_thinking_fp4, prec=fp4, TPxPPxCP=4x1x1, EP=4, attn_dp=True, bs=1, mnt=8448, msl=8232, kv=fp8/br=False/frac=0.75, moe=None, isl/osl=8192/1024wideep_kimi-k2-thinking-fp4_8k1k_ctx8_gen1_dep32_bs256_eplb416_mtp0_con8192_ccb-NIXL.yaml
+ +

Already-covered skips

+

No entries — QA's aggregated directory was empty before this run, so every unique ctx key translated into a brand-new generated YAML.

+ +

Excluded inputs

+

wideep_accuracy / wideep_stress (skipped by scope)

+
  • wideep_accuracy-deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_ccb-NIXL.yaml
  • wideep_accuracy-deepseek-r1-fp4_gpqa_diamond_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_ccb-NIXL.yaml
  • wideep_accuracy-kimi-k2-thinking-fp4_1k1k_ctx3_gen1_dep32_bs1024_eplb384_mtp0_ccb-NIXL.yaml
  • wideep_stress-deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_ccb-NIXL.yaml
+

Other read anomalies

+

No parse errors; every scanned YAML had a worker_config.ctx block.

+ +

Test list diff — tests/integration/test_lists/qa/llm_perf_multinode.txt

+

New # aggregated multi-node section inserted between the existing disagg section and the # wideep multi-node section. No existing line was modified; only these lines were appended.

+
+
++# aggregated multi-node
++# GB200 + GB300 supported cases
++perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]
++perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32-qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1-deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1-deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]
++perf/test_perf_sanity.py::test_e2e[aggr-glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]
++perf/test_perf_sanity.py::test_e2e[aggr-glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]
++perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32-gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32]
++perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32]
++perf/test_perf_sanity.py::test_e2e[aggr-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16-kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2-kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2]
++perf/test_perf_sanity.py::test_e2e[aggr-qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4-qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]
++perf/test_perf_sanity.py::test_e2e[aggr-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8]
++perf/test_perf_sanity.py::test_e2e[aggr-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1]
+ + + diff --git a/tests/integration/defs/perf/test_perf_sanity.py b/tests/integration/defs/perf/test_perf_sanity.py index 4a24d6f748d3..2b89d7557c77 100644 --- a/tests/integration/defs/perf/test_perf_sanity.py +++ b/tests/integration/defs/perf/test_perf_sanity.py @@ -362,15 +362,8 @@ def to_match_keys(self) -> List[str]: "l_cp", "l_gpus_per_node", "l_max_batch_size", - "b_disable_overlap_scheduler", - "b_enable_chunked_prefill", "b_enable_attention_dp", - "b_enable_lm_head_tp_in_adp", "s_serving_backend", - # attention_dp_config - "b_attention_dp_balance", - # cuda_graph_config - "b_enable_cuda_graph", # kv_cache_config "s_kv_cache_dtype", # cache_transceiver_config diff --git a/tests/integration/test_lists/qa/llm_perf_multinode.txt b/tests/integration/test_lists/qa/llm_perf_multinode.txt index cf9e938c8153..26d52be5d524 100644 --- a/tests/integration/test_lists/qa/llm_perf_multinode.txt +++ b/tests/integration/test_lists/qa/llm_perf_multinode.txt @@ -1,94 +1,103 @@ # disagg multi-node -# GB200 + GB300 supported cases -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL] +# Qwen backend-compare cases (GB300 only) + +# Qwen3-235B-A22B-FP4 gen4 sweep: keep con=8 (knee), 32 (max) +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL] + +# Qwen3-235B-A22B-FP8 gen1 sweep: keep con=8, 36 (max for this group) +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX] + +# Deepseek-r1 backend-compare cases (GB200 only) + +# deepseek-r1-fp4 1k1k ctx1 gen1 dep32 (unique ctx/gen shape vs gen4): keep both backends at con=1024 perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL] + +# deepseek-r1-fp4 1k1k ctx1 gen4 mtp0 sweep: keep con=8, 32 +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-NIXL] + +# deepseek-r1-fp4 1k1k ctx1 gen4 mtp3 sweep: keep con=8, 32 (mtp3 is distinct from mtp0 above) +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX] + +# deepseek-r1-fp4 1k1k ctx2 gen1 dep16 (unique ctx/gen shape): keep both backends at con=2048 perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX] + +# deepseek-r1-fp4 8k1k ctx1 gen1 dep32 (UCX-only, unique ctx/gen shape) perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL] + +# deepseek-r1-fp4 8k1k ctx1 gen3 bs16 sweep: keep con=8, 16 (max for this group) +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL] + +# deepseek-r1-fp4 8k1k ctx1 gen3 bs32 sweep: keep con=8, 32 (bs32 is distinct from bs16 above) +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] -perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-gen_only-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX] + +# deepseek-r1-fp4 8k1k ctx6 gen1 dep16 (unique ctx/gen shape): keep both backends perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX] + +# deepseek-r1-fp4 8k1k ctx8 gen1 dep32 (unique ctx/gen shape): keep both backends perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX] -# GB200 supported cases + +# dev-ported cases - GB200 + GB300 supported cases (from tests/scripts/perf-sanity/disaggregated/; see perf_test_sync_report.html) +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep4_bs512_eplb0_mtp1_con2048_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs64_eplb288_mtp1_con2048_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs8_eplb0_mtp3_con256_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs128_eplb256_mtp0_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_1k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp1_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_1k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_8k1k_ctx1_dep4_gen1_dep8_bs128_eplb0_mtp1_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_8k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-glm-5-fp4_8k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con2048_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con512_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_tp4_bs256_eplb0_mtp0_con64_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con128_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con4_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_dep2_bs512_eplb0_mtp0_con512_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_dep8_bs128_eplb0_mtp0_con1024_ccb-UCX] +perf/test_perf_sanity.py::test_e2e[disagg-e2e-qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_tep4_bs64_eplb0_mtp0_con64_ccb-UCX] + +# GB200 supported cases (each has unique ctx/gen/pp/dep/mtp combo — keep all) perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp8_gen11_tep4_bs2_eplb0_mtp0_con2-Default] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp8_gen14_tep4_bs1_eplb0_mtp0_con1-Default] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default] @@ -107,7 +116,7 @@ perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx3_pp8_ge perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default] -# GB300 supported cases +# GB300 supported cases (each has unique ctx/gen/pp/dep/mtp combo — keep all) perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp4_gen13_tep4_bs1_eplb0_mtp0_con1-Default] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp4_gen5_tep4_bs4_eplb0_mtp0_con4-Default] perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx1_pp4_gen6_tep8_bs1_eplb0_mtp3_con1-Default] @@ -129,27 +138,48 @@ perf/test_perf_sanity.py::test_e2e[disagg-e2e-deepseek-r1-fp4_128k8k_ctx8_pp4_ge + +# aggregated multi-node +# GB200 + GB300 supported cases +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8] +perf/test_perf_sanity.py::test_e2e[aggr-ctx_only-kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1] + # wideep multi-node # GB200 + GB300 supported cases -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb288_mtp3_con512_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb288_mtp3_con1024_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb288_mtp3_con512_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb288_mtp3_con1024_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp1_con2048_ccb-NIXL] -# perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp1_con2048_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL_kv-reuse] -perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT] perf/test_perf_sanity.py::test_e2e[disagg-e2e-wideep_deepseek-r1-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL] -perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL] perf/test_perf_sanity.py::test_e2e[disagg-gen_only-wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT] diff --git a/tests/integration/test_lists/test-db/l0_a10.yml b/tests/integration/test_lists/test-db/l0_a10.yml index aec960736072..e514952c8811 100644 --- a/tests/integration/test_lists/test-db/l0_a10.yml +++ b/tests/integration/test_lists/test-db/l0_a10.yml @@ -23,6 +23,7 @@ l0_a10: - unittest/_torch/modeling/test_modeling_cohere2.py - unittest/_torch/modeling/test_nemotron_nano_preprocessing.py - unittest/_torch/modeling/test_modeling_parakeet.py + - unittest/_torch/modeling/test_modeling_radio.py - unittest/_torch/sampler/test_trtllm_sampler.py - unittest/_torch/executor/test_async_transfer_manager.py - unittest/_torch/executor/test_scheduler_serializable_output.py @@ -224,9 +225,6 @@ l0_a10: stage: post_merge backend: tensorrt tests: - - test_e2e.py::test_mistral_e2e[use_py_session---] - - test_e2e.py::test_mistral_e2e[use_cpp_session-remove_input_padding--] - - test_e2e.py::test_mistral_e2e[use_py_session-remove_input_padding--] - examples/test_bert.py::test_llm_bert_general[compare_hf-disable_remove_input_padding-disable_attention_plugin-disable_context_fmha-tp:1-pp:1-float32-BertModel-bert/bert-base-uncased] - examples/test_bert.py::test_llm_bert_general[compare_hf-enable_remove_input_padding-use_attention_plugin-enable_context_fmha-tp:1-pp:1-float16-RobertaModel-bert/roberta-base] - examples/test_bert.py::test_llm_bert_general[compare_hf-enable_remove_input_padding-disable_attention_plugin-disable_context_fmha-tp:1-pp:1-float16-RobertaForSequenceClassification-bert/twitter-roberta-base-emotion] diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index 3484e91a8e53..3f7e2f7aec9d 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -120,7 +120,12 @@ l0_b200: # ------------- MoE: FlashInfer & TRTLLM symbol collision tests --------------- - unittest/_torch/flashinfer/test_trtllm_flashinfer_symbol_collision.py # --- MoE end - - unittest/_torch/multimodal + - unittest/_torch/multimodal/test_mm_encoder_standalone.py + - unittest/_torch/multimodal/test_multimodal_runtime.py + - unittest/_torch/multimodal/test_find_num_image_tokens.py + - unittest/_torch/multimodal/test_fuse_input_embeds.py + - unittest/_torch/multimodal/test_external_embedding.py + - unittest/_torch/multimodal/test_share_multiparams.py - unittest/_torch/sampler - unittest/_torch/speculative - unittest/_torch/thop/parallel TIMEOUT (90) diff --git a/tests/integration/test_lists/test-db/l0_dgx_h200_perf_sanity.yml b/tests/integration/test_lists/test-db/l0_dgx_h200_perf_sanity.yml new file mode 100644 index 000000000000..a923653ae3ce --- /dev/null +++ b/tests/integration/test_lists/test-db/l0_dgx_h200_perf_sanity.yml @@ -0,0 +1,25 @@ +version: 0.0.1 +l0_dgx_h200_perf_sanity: +- condition: + ranges: + system_gpu_count: + gte: 8 + lte: 8 + wildcards: + gpu: + - '*h200*' + linux_distribution_name: ubuntu* + cpu: x86_64 + terms: + stage: post_merge + backend: pytorch + tests: + + + - perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-h200_nemotron-super-fp8_8k1k_con64_ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX] + - perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-h200_qwen3-235b-a22b-fp8_8k1k_con512_ctx1_tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT] + - perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-h200_qwen3-32b-fp8_4k1k_con128_ctx1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT] + + # - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-h200_nemotron-super-fp8_8k1k_con64_ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX] + # - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-h200_qwen3-235b-a22b-fp8_8k1k_con512_ctx1_tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT] + # - perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-h200_qwen3-32b-fp8_4k1k_con128_ctx1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT] diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 2de8dfab584f..e0ac0304db70 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -364,8 +364,6 @@ accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_cuda_graph_padding_4gpus[attention_dp=True-mtp_nextn=0] SKIP (https://nvbugs/6084447) accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_nvfp4[latency_moe_trtllm_attention_dp] SKIP (https://nvbugs/6084568) perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-UCX] SKIP (https://nvbugs/6088149) -perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep32_eplb0_mtp3_ccb-UCX] SKIP (https://nvbugs/6088149) -perf/test_perf_sanity.py::test_e2e[aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell-k25_thinking_fp4_dep8_32k8k] SKIP (https://nvbugs/6088149) accuracy/test_llm_api_pytorch.py::TestNemotronNas::test_auto_dtype_tp8 SKIP (https://nvbugs/6070857) accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[fp8-1-trtllm] SKIP (https://nvbugs/6094208) accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-1-trtllm] SKIP (https://nvbugs/6094208) diff --git a/tests/scripts/perf-sanity/disaggregated/h200_nemotron-super-fp8_8k1k_con64_ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX.yaml b/tests/scripts/perf-sanity/disaggregated/h200_nemotron-super-fp8_8k1k_con64_ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX.yaml new file mode 100644 index 000000000000..2499fb4cf7f7 --- /dev/null +++ b/tests/scripts/perf-sanity/disaggregated/h200_nemotron-super-fp8_8k1k_con64_ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX.yaml @@ -0,0 +1,91 @@ +metadata: + model_name: super_fp8 + precision: fp8 + model_dir_name: NVIDIA-Nemotron-3-Super-120B-A12B-FP8 + supported_gpus: + - H200 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k + # Native-target (Hopper) mirror of the Dynamo Nemotron-3-Super-FP8 TRT-LLM + # disagg deployment recipe: + # https://github.com/ai-dynamo/dynamo/tree/main/recipes/nemotron-3-super-fp8/trtllm/disagg +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: "--gres=gpu:8" + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 + streaming: true + concurrency_list: '64' + input_length: 8192 + output_length: 1024 + dataset_file: datasets/perf-ci/nemotron_super-8k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 8 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +worker_config: + gen: + print_iter_log: true + tensor_parallel_size: 2 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: true + max_batch_size: 16 + max_num_tokens: 8192 + trust_remote_code: true + cuda_graph_config: + enable_padding: true + max_batch_size: 16 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + moe_config: + backend: TRTLLM + cache_transceiver_config: + backend: UCX + disable_overlap_scheduler: false + ctx: + print_iter_log: true + tensor_parallel_size: 2 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: true + max_batch_size: 16 + max_num_tokens: 8192 + trust_remote_code: true + cuda_graph_config: + enable_padding: true + max_batch_size: 16 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + moe_config: + backend: TRTLLM + cache_transceiver_config: + backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml b/tests/scripts/perf-sanity/disaggregated/h200_qwen3-235b-a22b-fp8_8k1k_con512_ctx1_tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT.yaml similarity index 51% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml rename to tests/scripts/perf-sanity/disaggregated/h200_qwen3-235b-a22b-fp8_8k1k_con512_ctx1_tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT.yaml index 4526cc4dd51e..0bf12a3fe300 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml +++ b/tests/scripts/perf-sanity/disaggregated/h200_qwen3-235b-a22b-fp8_8k1k_con512_ctx1_tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT.yaml @@ -3,30 +3,32 @@ metadata: precision: fp8 model_dir_name: Qwen3-235B-A22B-FP8 supported_gpus: - - GB200 - - GB300 + - H200 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k + # Native-target (Hopper) mirror of the Dynamo Qwen3-235B-A22B-FP8 TRT-LLM + # disagg deployment recipe: + # https://github.com/ai-dynamo/dynamo/tree/main/recipes/qwen3-235b-a22b-fp8/trtllm/disagg slurm: script_file: disaggr_torch.slurm partition: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: --gres=gpu:4 + extra_args: "--gres=gpu:8" numa_bind: true benchmark: mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 streaming: true - concurrency_list: '16' - input_length: 1024 + concurrency_list: '512' + input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/qwen3_235b-8k1k-20480-ratio-1_for_serve.json hardware: - gpus_per_node: 4 + gpus_per_node: 8 num_ctx_servers: 1 num_gen_servers: 1 environment: @@ -36,55 +38,60 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 + worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" profiling: nsys_on: false accuracy: enable_accuracy_test: false worker_config: gen: + print_iter_log: true tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: false + moe_tensor_parallel_size: 1 pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: false + trust_remote_code: true + max_batch_size: 512 + max_num_tokens: 1024 + max_seq_len: 8192 cuda_graph_config: enable_padding: true - max_batch_size: 128 - print_iter_log: true + max_batch_size: 512 kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 + enable_block_reuse: false + free_gpu_memory_fraction: 0.95 dtype: fp8 moe_config: - backend: TRTLLM + backend: DEEPGEMM cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL + backend: DEFAULT disable_overlap_scheduler: false ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true + tensor_parallel_size: 2 + moe_expert_parallel_size: 1 + moe_tensor_parallel_size: 2 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: false + trust_remote_code: true + max_batch_size: 2 + max_num_tokens: 8192 + max_seq_len: 8192 + cuda_graph_config: + enable_padding: true + max_batch_size: 2 kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.7 dtype: fp8 moe_config: - backend: TRTLLM + backend: DEEPGEMM cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL + backend: DEFAULT + disable_overlap_scheduler: true diff --git a/tests/scripts/perf-sanity/disaggregated/h200_qwen3-32b-fp8_4k1k_con128_ctx1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT.yaml b/tests/scripts/perf-sanity/disaggregated/h200_qwen3-32b-fp8_4k1k_con128_ctx1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT.yaml new file mode 100644 index 000000000000..e7eb2ce64804 --- /dev/null +++ b/tests/scripts/perf-sanity/disaggregated/h200_qwen3-32b-fp8_4k1k_con128_ctx1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT.yaml @@ -0,0 +1,89 @@ +metadata: + model_name: qwen3_32b_fp8 + precision: fp8 + model_dir_name: Qwen3-32B-FP8 + supported_gpus: + - H200 + script_file: disaggr_torch.slurm + benchmark_type: 4k1k + # Native-target (Hopper) mirror of the Dynamo Qwen3-32B-FP8 TRT-LLM disagg + # deployment recipe: + # https://github.com/ai-dynamo/dynamo/tree/main/recipes/qwen3-32b-fp8/trtllm/disagg +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: "--gres=gpu:8" + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: false + multi_round: 10 + benchmark_ratio: 0.0 + streaming: true + concurrency_list: '128' + input_length: 4096 + output_length: 1024 + dataset_file: datasets/perf-ci/qwen3_32b-4k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 8 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=True" + server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +worker_config: + gen: + print_iter_log: true + tensor_parallel_size: 2 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: false + trust_remote_code: true + max_batch_size: 128 + max_num_tokens: 7808 + max_seq_len: 7808 + cuda_graph_config: + enable_padding: true + max_batch_size: 128 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + cache_transceiver_config: + backend: DEFAULT + disable_overlap_scheduler: false + ctx: + print_iter_log: true + tensor_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + enable_chunked_prefill: false + trust_remote_code: true + max_batch_size: 1 + max_num_tokens: 7808 + max_seq_len: 7808 + cuda_graph_config: + enable_padding: true + max_batch_size: 256 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + cache_transceiver_config: + backend: DEFAULT + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1.yaml new file mode 100644 index 000000000000..9ad025cae646 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1.yaml @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB300 + benchmark_type: 128k8k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1 + tensor_parallel_size: 1 + pipeline_parallel_size: 4 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 1 + max_num_tokens: 131104 + max_seq_len: 131104 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.4 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-r1-fp4_128k8k_ctx_pp4_fp4_bs1_client + concurrency_list: '1' + input_length: 131072 + output_length: 8192 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1.yaml new file mode 100644 index 000000000000..03569636b03c --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 128k8k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1 + tensor_parallel_size: 1 + pipeline_parallel_size: 8 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 1 + max_num_tokens: 131104 + max_seq_len: 131104 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.4 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-r1-fp4_128k8k_ctx_pp8_fp4_bs1_client + concurrency_list: '1' + input_length: 131072 + output_length: 8192 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml new file mode 100644 index 000000000000..7e92da3fd498 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4608 + max_seq_len: 2251 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml new file mode 100644 index 000000000000..8d4c3e0e6e1f --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4608 + max_seq_len: 2251 + kv_cache_config: + enable_block_reuse: true + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2.yaml new file mode 100644 index 000000000000..605883eac74e --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4480 + max_seq_len: 2176 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v2_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml new file mode 100644 index 000000000000..dd97b748db61 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 8448 + max_seq_len: 9419 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml new file mode 100644 index 000000000000..3215e5f0828a --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 8448 + max_seq_len: 9423 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml new file mode 100644 index 000000000000..e1c8d9e2120d --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_r1_0528_fp4_v2 + precision: fp4 + model_dir_name: DeepSeek-R1-0528-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 2 + max_num_tokens: 16896 + max_seq_len: 9256 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: deepseek-r1-fp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml new file mode 100644 index 000000000000..e7c16576cc8b --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs16_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml new file mode 100644 index 000000000000..7189e7a0706b --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4608 + max_seq_len: 2251 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml new file mode 100644 index 000000000000..34de24ebae79 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4480 + max_seq_len: 2176 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-v32-fp4-v2_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_v1_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1.yaml new file mode 100644 index 000000000000..0de8c59ba40b --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 32k4k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 32784 + max_seq_len: 32832 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.4 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: deepseek-v32-fp4-v2_32k4k_ctx_tp4_ep4_fp4_attndp_bs1_client + concurrency_list: '1' + input_length: 32768 + output_length: 4096 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml new file mode 100644 index 000000000000..8e6ce85dec57 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 8448 + max_seq_len: 9423 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml new file mode 100644 index 000000000000..e9213775e9b0 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 8448 + max_seq_len: 9419 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_v1_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml new file mode 100644 index 000000000000..c6bd10873d07 --- /dev/null +++ b/tests/scripts/perf/aggregated/deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 2 + max_num_tokens: 16384 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: deepseek-v32-fp4-v2_8k1k_ctx_tp4_ep4_fp4_attndp_bs2_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml b/tests/scripts/perf/aggregated/glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml new file mode 100644 index 000000000000..8c854e42cd11 --- /dev/null +++ b/tests/scripts/perf/aggregated/glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: glm_5_nvfp4 + precision: fp4 + model_dir_name: GLM-5-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 16 + max_num_tokens: 16384 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: glm-5-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml b/tests/scripts/perf/aggregated/glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml new file mode 100644 index 000000000000..3b42bd7793c2 --- /dev/null +++ b/tests/scripts/perf/aggregated/glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: glm_5_nvfp4 + precision: fp4 + model_dir_name: GLM-5-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 2 + max_num_tokens: 16384 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: glm-5-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32.yaml b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32.yaml new file mode 100644 index 000000000000..99ac89858945 --- /dev/null +++ b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32 + tensor_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 32 + max_num_tokens: 20000 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: gpt-oss-120b-fp4_1k1k_ctx_tp1_fp4_bs32_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32.yaml b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32.yaml new file mode 100644 index 000000000000..8c186f930550 --- /dev/null +++ b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32 + tensor_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 32 + max_num_tokens: 20000 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs32_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64.yaml b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64.yaml new file mode 100644 index 000000000000..e4c907b07030 --- /dev/null +++ b/tests/scripts/perf/aggregated/gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64 + tensor_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 64 + max_num_tokens: 20000 + max_seq_len: 9000 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: gpt-oss-120b-fp4_8k1k_ctx_tp1_fp4_bs64_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml new file mode 100644 index 000000000000..5a11e6c1299a --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 16 + max_num_tokens: 8192 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8.yaml b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8.yaml new file mode 100644 index 000000000000..0cd0dc0f49eb --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 8 + max_num_tokens: 8448 + max_seq_len: 1044 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: kimi-k2-thinking-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs8_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml new file mode 100644 index 000000000000..f4fd6d767e49 --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 1 + max_num_tokens: 8448 + max_seq_len: 8232 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs1_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml new file mode 100644 index 000000000000..55d10b9b49f5 --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 2 + max_num_tokens: 8768 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: kimi-k2-thinking-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml b/tests/scripts/perf/aggregated/kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml new file mode 100644 index 000000000000..88c67673627a --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k25_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2.5-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 16 + max_num_tokens: 8192 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: kimi-k25-nvfp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs16_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml b/tests/scripts/perf/aggregated/kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml new file mode 100644 index 000000000000..46c535d255b7 --- /dev/null +++ b/tests/scripts/perf/aggregated/kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: k25_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2.5-NVFP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 2 + max_num_tokens: 8768 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: kimi-k25-nvfp4_8k1k_ctx_tp4_ep4_fp4_attndp_bs2_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml new file mode 100644 index 000000000000..b62fd39b4f0e --- /dev/null +++ b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: qwen3_235b_a22b_fp4 + precision: fp4 + model_dir_name: Qwen3-235B-A22B-FP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: true + max_batch_size: 4 + max_num_tokens: 4608 + max_seq_len: 2251 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true +client_configs: +- name: qwen3-235b-a22b-fp4_1k1k_ctx_tp4_ep4_fp4_attndp_bs4_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4.yaml b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4.yaml new file mode 100644 index 000000000000..e27658707ab8 --- /dev/null +++ b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: qwen3_235b_a22b_fp4 + precision: fp4 + model_dir_name: Qwen3-235B-A22B-FP4 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 8k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4 + tensor_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 4 + max_num_tokens: 32768 + max_seq_len: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: CUTEDSL +client_configs: +- name: qwen3-235b-a22b-fp4_8k1k_ctx_tp1_fp4_bs4_client + concurrency_list: '1' + input_length: 8192 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32.yaml b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32.yaml new file mode 100644 index 000000000000..47242c99301c --- /dev/null +++ b/tests/scripts/perf/aggregated/qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +metadata: + model_name: qwen3_235b_a22b_fp8 + precision: fp8 + model_dir_name: Qwen3-235B-A22B-FP8 + supported_gpus: + - GB200 + - GB300 + benchmark_type: 1k1k +slurm: + partition: + account: + job_time: 01:00:00 + job_name: agg-ctx-only + extra_args: --gres=gpu:4 + numa_bind: true +hardware: + gpus_per_node: 4 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 + TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false +server_configs: +- name: qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32 + tensor_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + moe_expert_parallel_size: 4 + enable_attention_dp: false + max_batch_size: 32 + max_num_tokens: 2048 + max_seq_len: 2051 + kv_cache_config: + enable_block_reuse: true + free_gpu_memory_fraction: 0.7 + dtype: fp8 + print_iter_log: true + disable_overlap_scheduler: true + moe_config: + backend: TRTLLM +client_configs: +- name: qwen3-235b-a22b-fp8_1k1k_ctx_tp4_ep4_fp8_bs32_client + concurrency_list: '1' + input_length: 1024 + output_length: 1024 + dataset_file: diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml deleted file mode 100644 index aef1a1cd8c52..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-NIXL.yaml +++ /dev/null @@ -1,105 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1024' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 256 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: WIDEEP - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX.yaml deleted file mode 100644 index 2c4d11208df7..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con1024_ccb-UCX.yaml +++ /dev/null @@ -1,105 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1024' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 256 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: WIDEEP - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL.yaml deleted file mode 100644 index 87939ffce62e..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-NIXL.yaml +++ /dev/null @@ -1,105 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '512' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 256 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: WIDEEP - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml deleted file mode 100644 index 16f9cdbfa56c..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml deleted file mode 100644 index b9b208e78c8e..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml deleted file mode 100644 index a12241eec623..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml index 2688bc7fb2c2..a44f269592b1 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml @@ -3,7 +3,6 @@ metadata: precision: fp4 model_dir_name: Qwen3-235B-A22B-FP4 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml index c2cf88e83fc4..3c4d98dea2b9 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml @@ -3,7 +3,6 @@ metadata: precision: fp4 model_dir_name: Qwen3-235B-A22B-FP4 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml deleted file mode 100644 index 9f1ccb70faa9..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp4 - precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml index 78cde1a8cf0c..37e6b043db93 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml @@ -3,7 +3,6 @@ metadata: precision: fp4 model_dir_name: Qwen3-235B-A22B-FP4 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml index 9a857429126b..8b80da04b31c 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml @@ -3,7 +3,6 @@ metadata: precision: fp4 model_dir_name: Qwen3-235B-A22B-FP4 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml deleted file mode 100644 index c301e3fba76e..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml deleted file mode 100644 index e1e660432a3d..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml deleted file mode 100644 index 106ff6e8c0a1..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml deleted file mode 100644 index 22972782d2de..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL.yaml index f8f454f1f8ec..1f3a665e9170 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-NIXL.yaml @@ -3,7 +3,6 @@ metadata: precision: fp8 model_dir_name: Qwen3-235B-A22B-FP8 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX.yaml index 5819abaf3ec1..e22a5fa0d875 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con36_ccb-UCX.yaml @@ -3,7 +3,6 @@ metadata: precision: fp8 model_dir_name: Qwen3-235B-A22B-FP8 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml deleted file mode 100644 index 7beae38501ff..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml deleted file mode 100644 index 52ed89a6724e..000000000000 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml +++ /dev/null @@ -1,90 +0,0 @@ -metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 1 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 - cuda_graph_config: - enable_padding: true - max_batch_size: 128 - print_iter_log: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - disable_overlap_scheduler: false - ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 2048 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml index 9b43900bcfc1..c24b7d58b570 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml @@ -3,7 +3,6 @@ metadata: precision: fp8 model_dir_name: Qwen3-235B-A22B-FP8 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml index ec6e8c741333..c7b1388e94f0 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml @@ -3,7 +3,6 @@ metadata: precision: fp8 model_dir_name: Qwen3-235B-A22B-FP8 supported_gpus: - - GB200 - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default.yaml index 1cc2b23136b0..63c5f9679a39 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep16_bs1_eplb0_mtp3_con1-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep8_bs4_eplb0_mtp2_con4-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep8_bs4_eplb0_mtp2_con4-Default.yaml index c8ac64526c98..f87316349a6e 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep8_bs4_eplb0_mtp2_con4-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx1_pp8_gen1_dep8_bs4_eplb0_mtp2_con4-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep16_bs8_eplb0_mtp0_con8-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep16_bs8_eplb0_mtp0_con8-Default.yaml index 8d5b7bf31309..233236dab6ac 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep16_bs8_eplb0_mtp0_con8-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep16_bs8_eplb0_mtp0_con8-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep32_bs2_eplb0_mtp0_con2-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep32_bs2_eplb0_mtp0_con2-Default.yaml index 567a6eadf440..025b06668611 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep32_bs2_eplb0_mtp0_con2-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx2_pp8_gen1_dep32_bs2_eplb0_mtp0_con2-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp4_gen1_dep8_bs16_eplb0_mtp1_con128-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp4_gen1_dep8_bs16_eplb0_mtp1_con128-Default.yaml index ab39577367ed..8da543d33591 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp4_gen1_dep8_bs16_eplb0_mtp1_con128-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp4_gen1_dep8_bs16_eplb0_mtp1_con128-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs16_eplb0_mtp0_con16-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs16_eplb0_mtp0_con16-Default.yaml index 5ad97f542f29..3798bdda4106 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs16_eplb0_mtp0_con16-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs16_eplb0_mtp0_con16-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs8_eplb0_mtp2_con8-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs8_eplb0_mtp2_con8-Default.yaml index 10b8cfed99eb..3363babef873 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs8_eplb0_mtp2_con8-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep16_bs8_eplb0_mtp2_con8-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default.yaml index cbade5120619..8ec84d190de4 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs2_eplb0_mtp3_con2-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default.yaml index 3af38a83d219..3f471e2dd724 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx3_pp8_gen1_dep32_bs4_eplb0_mtp0_con4-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs16_eplb0_mtp0_con256-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs16_eplb0_mtp0_con256-Default.yaml index e4d8dc7346c5..ea953108c630 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs16_eplb0_mtp0_con256-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs16_eplb0_mtp0_con256-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs8_eplb0_mtp3_con128-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs8_eplb0_mtp3_con128-Default.yaml index 8d75cbb5c2a9..5ad6b9eef127 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs8_eplb0_mtp3_con128-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep16_bs8_eplb0_mtp3_con128-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs2_eplb0_mtp3_con64-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs2_eplb0_mtp3_con64-Default.yaml index aeb4581a3b95..a8b2aa34aa22 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs2_eplb0_mtp3_con64-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs2_eplb0_mtp3_con64-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs4_eplb0_mtp0_con128-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs4_eplb0_mtp0_con128-Default.yaml index 2ffde0adda0a..1a4e1e9aacd1 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs4_eplb0_mtp0_con128-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx5_pp4_gen1_dep32_bs4_eplb0_mtp0_con128-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs16_eplb0_mtp1_con256-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs16_eplb0_mtp1_con256-Default.yaml index 67e28bdcc347..8720da19d5d1 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs16_eplb0_mtp1_con256-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs16_eplb0_mtp1_con256-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs32_eplb0_mtp0_con512-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs32_eplb0_mtp0_con512-Default.yaml index 58fa55138cd5..a79e632bbcbc 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs32_eplb0_mtp0_con512-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx7_pp4_gen1_dep16_bs32_eplb0_mtp0_con512-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep16_bs32_eplb0_mtp1_con512-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep16_bs32_eplb0_mtp1_con512-Default.yaml index fc316de34aa5..f9f3dd0220f0 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep16_bs32_eplb0_mtp1_con512-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep16_bs32_eplb0_mtp1_con512-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs4_eplb0_mtp3_con128-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs4_eplb0_mtp3_con128-Default.yaml index 7857a0b22339..59dc5c0c11f7 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs4_eplb0_mtp3_con128-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs4_eplb0_mtp3_con128-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp0_con256-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp0_con256-Default.yaml index 161ef82e7c40..7769d45430c5 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp0_con256-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp0_con256-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp3_con256-Default.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp3_con256-Default.yaml index 42f6c6b9bca8..86806c00a2c6 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp3_con256-Default.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_128k8k_ctx8_pp4_gen1_dep32_bs8_eplb0_mtp3_con256-Default.yaml @@ -69,7 +69,7 @@ worker_config: free_gpu_memory_fraction: 0.4 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 131104 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL.yaml index 2ce6a987e6f6..3e0d4b0f9ac6 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -37,8 +36,9 @@ environment: build_wheel: false trtllm_wheel_path: '' work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX.yaml index f46e66f455ec..63776403f756 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb0_mtp0_con1024_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml deleted file mode 100644 index 7b0b8b627f26..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml deleted file mode 100644 index ed1783847a3d..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml deleted file mode 100644 index ba1e40738ce7..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml deleted file mode 100644 index 13d3d5515089..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml deleted file mode 100644 index 90ef56d11751..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml deleted file mode 100644 index 3dcbecb8fdcc..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml index 1947ac703aed..f29c39993271 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml index de7b1291bb46..b293261302e2 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml deleted file mode 100644 index a46d642607aa..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml index bc4618fe923d..682dd64f972f 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml index 4bda05c3487e..526fb9a9f4fb 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-NIXL.yaml deleted file mode 100644 index 9829e43592ad..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-NIXL.yaml deleted file mode 100644 index 9d75b2608e96..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-NIXL.yaml deleted file mode 100644 index 6689708c5bbe..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL.yaml index 069fef04d023..b27290d104be 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX.yaml index d36db50a2bab..8af57bc87b7f 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con32_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-NIXL.yaml deleted file mode 100644 index faef25215337..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 1k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 1024 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 4 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.85 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL.yaml index f818041c76c0..2a8d20005892 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX.yaml index ed8490bf114e..4ee5b5761a25 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con8_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL.yaml index 3cd56a5e25ae..5f72ddd4f205 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX.yaml index 3c9278738e4c..d0ec3f17ce3c 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp3_con2048_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yaml index 2afc97ccceb4..d6f1a5882bf7 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen1_dep32_bs128_eplb0_mtp3_con1024_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 1k1k config_index: -1 @@ -14,7 +13,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: gen_only @@ -37,8 +36,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL.yaml index 34baacf0729f..9f514617113d 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX.yaml index b1258b3ff451..965cbd6e49ad 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con16_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-NIXL.yaml deleted file mode 100644 index ed47d2c1c7c7..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-NIXL.yaml deleted file mode 100644 index cf6db3500f06..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-NIXL.yaml deleted file mode 100644 index 8e6cbb5e484d..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,106 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL.yaml index fcc652a4b1a3..77c2779779b7 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX.yaml index 68005a14da7e..cd5d03cae8a1 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con8_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml deleted file mode 100644 index 18e10cbdb5ad..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml deleted file mode 100644 index 239e1c608895..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '16' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: UCX diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml deleted file mode 100644 index 1116c0051979..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '1' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml deleted file mode 100644 index 29e92e4b48c9..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '2' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml index a7d9ec5458ae..36381b4f7e41 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml index 193e2d19d3f0..9dfd7ce3b880 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con32_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml deleted file mode 100644 index e26e9a057d54..000000000000 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-NIXL.yaml +++ /dev/null @@ -1,100 +0,0 @@ -metadata: - model_name: deepseek_r1_0528_fp4_v2 - precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 - supported_gpus: - - GB200 - - GB300 - script_file: disaggr_torch.slurm - benchmark_type: 8k1k -slurm: - script_file: disaggr_torch.slurm - partition: - account: - job_time: 02:00:00 - job_name: unified-benchmark - extra_args: --gres=gpu:4 - numa_bind: true -benchmark: - mode: e2e - use_nv_sa_benchmark: true - multi_round: 8 - benchmark_ratio: 0.8 - streaming: true - concurrency_list: '4' - input_length: 8192 - output_length: 1024 - dataset_file: -hardware: - gpus_per_node: 4 - num_ctx_servers: 1 - num_gen_servers: 3 -environment: - container_mount: - container_image: - model_path: - trtllm_repo: '' - build_wheel: false - work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" - server_env_var: TRTLLM_SERVER_DISABLE_GC=1 -profiling: - nsys_on: false -accuracy: - enable_accuracy_test: false -worker_config: - gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 - cuda_graph_config: - enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.7 - dtype: fp8 - moe_config: - backend: TRTLLM - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL - ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 - print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true - kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.75 - dtype: fp8 - cache_transceiver_config: - max_tokens_in_buffer: 8448 - backend: NIXL diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml index 4ba41f92cd31..271026089f98 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml index 033dfdf378fc..fe863db2cff1 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con8_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -16,9 +15,9 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: e2e + mode: gen_only use_nv_sa_benchmark: true - multi_round: 8 + multi_round: 1 benchmark_ratio: 0.8 streaming: true concurrency_list: '8' @@ -36,7 +35,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 + ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL.yaml index a13f4dca63eb..2cde3841869f 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX.yaml index 1b36fa91a381..531f2ddf2b83 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb0_mtp0_con1024_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml index 29fa7e3eb95c..89c5118eeb6a 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml index e9aa9d4ebb6b..4b05df38d340 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml @@ -4,7 +4,6 @@ metadata: model_dir_name: DeepSeek-R1-0528-FP4-v2 supported_gpus: - GB200 - - GB300 script_file: disaggr_torch.slurm benchmark_type: 8k1k slurm: @@ -13,7 +12,7 @@ slurm: account: job_time: 02:00:00 job_name: unified-benchmark - extra_args: "--gres=gpu:4" + extra_args: --gres=gpu:4 numa_bind: true benchmark: mode: e2e @@ -36,8 +35,9 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes MIMALLOC_PURGE_DELAY=0" - server_env_var: "TRTLLM_SERVER_DISABLE_GC=1" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + MIMALLOC_PURGE_DELAY=0 + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: diff --git a/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml new file mode 100644 index 000000000000..0e61dfde52f5 --- /dev/null +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml @@ -0,0 +1,111 @@ +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 1k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '1024' + input_length: 1024 + output_length: 1024 + dataset_file: datasets/perf-ci/deepseek_v32-1k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 128 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 32 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 256 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: MTP + num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core + ctx: + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep4_bs512_eplb0_mtp1_con2048_ccb-UCX.yaml similarity index 52% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-NIXL.yaml rename to tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep4_bs512_eplb0_mtp1_con2048_ccb-UCX.yaml index b05cf6fca544..59b85cdfd52f 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_dep4_bs512_eplb0_mtp1_con2048_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: deepseek_v32_fp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: DeepSeek-V3.2-FP4-v2 supported_gpus: - GB200 - GB300 @@ -24,10 +24,10 @@ benchmark: concurrency_list: '2048' input_length: 1024 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/deepseek_v32-1k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 - num_ctx_servers: 2 + num_ctx_servers: 1 num_gen_servers: 1 environment: container_mount: @@ -36,70 +36,73 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true + print_iter_log: true + max_batch_size: 512 + max_num_tokens: 1024 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 pipeline_parallel_size: 1 - max_batch_size: 128 - max_num_tokens: 256 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 512 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.8 dtype: fp8 + tokens_per_block: 64 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 1 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 1 + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml similarity index 59% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml index e147acb03f5c..419f74f932e3 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con2_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_1k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: deepseek_v32_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: DeepSeek-V3.2-FP4-v2 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2' + concurrency_list: '1' input_length: 1024 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/deepseek_v32-1k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,72 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 1 + max_num_tokens: 512 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 1 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.9 dtype: fp8 + tokens_per_block: 64 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core allreduce_strategy: MNNVL ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs64_eplb288_mtp1_con2048_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs64_eplb288_mtp1_con2048_ccb-UCX.yaml new file mode 100644 index 000000000000..1929bb2b1ae2 --- /dev/null +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs64_eplb288_mtp1_con2048_ccb-UCX.yaml @@ -0,0 +1,115 @@ +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 32k4k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '2048' + input_length: 32768 + output_length: 4096 + dataset_file: datasets/perf-ci/deepseek_v32-32k4k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 64 + max_num_tokens: 128 + max_input_len: 32784 + max_seq_len: 40960 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 64 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.5 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 288 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 32832 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: MTP + num_nextn_predict_layers: 1 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core + ctx: + print_iter_log: true + max_batch_size: 1 + max_num_tokens: 32784 + max_input_len: 32784 + max_seq_len: 32832 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.4 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 32832 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs8_eplb0_mtp3_con256_ccb-UCX.yaml similarity index 50% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs8_eplb0_mtp3_con256_ccb-UCX.yaml index 1be6cd5d8399..16c9bdba25a2 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep16_bs64_eplb0_mtp3_con512_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_dep32_bs8_eplb0_mtp3_con256_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: deepseek_v32_fp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: DeepSeek-V3.2-FP4-v2 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 32k4k slurm: script_file: disaggr_torch.slurm partition: @@ -21,10 +21,10 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '512' - input_length: 1024 - output_length: 1024 - dataset_file: + concurrency_list: '256' + input_length: 32768 + output_length: 4096 + dataset_file: datasets/perf-ci/deepseek_v32-32k4k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 @@ -36,70 +36,77 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true + print_iter_log: true + max_batch_size: 8 + max_num_tokens: 64 + max_input_len: 32784 + max_seq_len: 40960 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 256 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 8 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.5 dtype: fp8 + tokens_per_block: 64 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32832 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 1 + max_num_tokens: 32784 + max_input_len: 32784 + max_seq_len: 32832 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.4 dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32832 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml similarity index 54% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml index 0c413f332660..c012b9f03a0a 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con2_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_32k4k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: deepseek_v32_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: DeepSeek-V3.2-FP4-v2 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 8k1k + benchmark_type: 32k4k slurm: script_file: disaggr_torch.slurm partition: @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2' - input_length: 8192 - output_length: 1024 - dataset_file: + concurrency_list: '1' + input_length: 32768 + output_length: 4096 + dataset_file: datasets/perf-ci/deepseek_v32-32k4k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,77 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 1 + max_num_tokens: 8 + max_input_len: 32784 + max_seq_len: 40960 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: false + enable_lm_head_tp_in_adp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 1 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.25 dtype: fp8 + tokens_per_block: 64 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 32832 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core allreduce_strategy: MNNVL ctx: + print_iter_log: true max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 + max_num_tokens: 32784 + max_input_len: 32784 + max_seq_len: 32832 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.4 dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 32832 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs128_eplb256_mtp0_con4096_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs128_eplb256_mtp0_con4096_ccb-UCX.yaml new file mode 100644 index 000000000000..56f804bb946f --- /dev/null +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs128_eplb256_mtp0_con4096_ccb-UCX.yaml @@ -0,0 +1,107 @@ +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '4096' + input_length: 8192 + output_length: 1024 + dataset_file: datasets/perf-ci/deepseek_v32-8k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 128 + max_num_tokens: 128 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 128 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 256 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core + ctx: + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml new file mode 100644 index 000000000000..4fd333ae2c3f --- /dev/null +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb256_mtp3_con1024_ccb-UCX.yaml @@ -0,0 +1,111 @@ +metadata: + model_name: deepseek_v32_fp4 + precision: fp4 + model_dir_name: DeepSeek-V3.2-FP4-v2 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '1024' + input_length: 8192 + output_length: 1024 + dataset_file: datasets/perf-ci/deepseek_v32-8k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 128 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 32 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.75 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 256 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: MTP + num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core + ctx: + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml similarity index 59% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml index 49fb3db339f0..b3f941ab5340 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con4_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/deepseek-v32-fp4_8k1k_ctx1_dep4_gen1_tep8_bs1_eplb0_mtp3_con1_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: deepseek_v32_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: DeepSeek-V3.2-FP4-v2 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '4' + concurrency_list: '1' input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/deepseek_v32-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,72 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 1 + max_num_tokens: 512 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 1 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.9 dtype: fp8 + tokens_per_block: 64 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 + nvfp4_gemm_config: + allowed_backends: + - cutlass + - cublaslt + - cutedsl + - cuda_core allreduce_strategy: MNNVL ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.6 dtype: fp8 + tokens_per_block: 64 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml similarity index 65% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml index e675219be991..ed5803cc6d38 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 @@ -36,70 +36,65 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 128 tensor_parallel_size: 32 moe_expert_parallel_size: 32 - enable_attention_dp: true pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 16 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.6 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp1_con4096_ccb-UCX.yaml similarity index 58% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp1_con4096_ccb-UCX.yaml index f42cbd4138d3..bc532bfcb4b6 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con16_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp1_con4096_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '16' + concurrency_list: '4096' input_length: 1024 output_length: 1024 dataset_file: hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,65 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 512 + max_num_tokens: 1024 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 512 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP - num_nextn_predict_layers: 3 - allreduce_strategy: MNNVL + num_nextn_predict_layers: 1 + num_postprocess_workers: 4 + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml similarity index 66% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml index ae464650ee1f..77e42038fcff 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con1_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_1k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 @@ -28,7 +28,7 @@ benchmark: hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,38 +36,28 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 128 + max_num_tokens: 512 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 128 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.9 @@ -75,32 +65,35 @@ worker_config: moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 allreduce_strategy: MNNVL ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml similarity index 62% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml index 255efb49512d..409e8c4d4c84 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep32_bs16_eplb0_mtp3_con512_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -22,7 +22,7 @@ benchmark: benchmark_ratio: 0.8 streaming: true concurrency_list: '512' - input_length: 1024 + input_length: 8192 output_length: 1024 dataset_file: hardware: @@ -36,70 +36,65 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 128 tensor_parallel_size: 32 moe_expert_parallel_size: 32 - enable_attention_dp: true pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 16 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.6 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 - backend: NIXL - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep8_bs128_eplb0_mtp1_con1024_ccb-UCX.yaml similarity index 59% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep8_bs128_eplb0_mtp1_con1024_ccb-UCX.yaml index 49f07a087b6e..a35b1441d693 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx2_gen1_dep16_bs128_eplb0_mtp1_con2048_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_dep8_bs128_eplb0_mtp1_con1024_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -21,13 +21,13 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2048' - input_length: 1024 + concurrency_list: '1024' + input_length: 8192 output_length: 1024 dataset_file: hardware: gpus_per_node: 4 - num_ctx_servers: 2 + num_ctx_servers: 1 num_gen_servers: 1 environment: container_mount: @@ -36,70 +36,65 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 16 - moe_expert_parallel_size: 16 - enable_attention_dp: true - pipeline_parallel_size: 1 + print_iter_log: true max_batch_size: 128 max_num_tokens: 256 - max_seq_len: 2251 + tensor_parallel_size: 8 + moe_expert_parallel_size: 8 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 128 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 1 + num_postprocess_workers: 4 + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 1 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml similarity index 65% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml index 0ab9b2e25dce..9b3676e407f7 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs16_eplb0_mtp3_con1_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/glm-5-fp4_8k1k_ctx1_dep4_gen1_tep8_bs128_eplb0_mtp3_con1_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: glm_5_nvfp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GLM-5-NVFP4 supported_gpus: - GB200 - GB300 @@ -28,7 +28,7 @@ benchmark: hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,64 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 128 + max_num_tokens: 512 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 128 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.85 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - speculative_config: + disable_overlap_scheduler: true + speculative_config: &id001 decoding_type: MTP num_nextn_predict_layers: 3 + num_postprocess_workers: 4 + stream_interval: 20 allreduce_strategy: MNNVL ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 16384 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 diff --git a/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con2048_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con2048_ccb-UCX.yaml new file mode 100644 index 000000000000..cb8a4d24a11b --- /dev/null +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con2048_ccb-UCX.yaml @@ -0,0 +1,97 @@ +metadata: + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 1k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '2048' + input_length: 1024 + output_length: 1024 + dataset_file: datasets/perf-ci/gpt_oss_120b-1k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 1536 + max_num_tokens: 20000 + tensor_parallel_size: 2 + moe_expert_parallel_size: 2 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + attention_dp_config: + batching_wait_iters: 0 + enable_balance: true + timeout_iters: 60 + cuda_graph_config: + enable_padding: true + max_batch_size: 1536 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + moe_config: + backend: TRTLLM + cache_transceiver_config: + max_tokens_in_buffer: 1024 + backend: UCX + disable_overlap_scheduler: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 1024 + backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con512_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con512_ccb-UCX.yaml new file mode 100644 index 000000000000..fabc69a9f834 --- /dev/null +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_dep2_bs1536_eplb0_mtp0_con512_ccb-UCX.yaml @@ -0,0 +1,97 @@ +metadata: + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 1k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '512' + input_length: 1024 + output_length: 1024 + dataset_file: datasets/perf-ci/gpt_oss_120b-1k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 1536 + max_num_tokens: 20000 + tensor_parallel_size: 2 + moe_expert_parallel_size: 2 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + attention_dp_config: + batching_wait_iters: 0 + enable_balance: true + timeout_iters: 60 + cuda_graph_config: + enable_padding: true + max_batch_size: 1536 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + moe_config: + backend: TRTLLM + cache_transceiver_config: + max_tokens_in_buffer: 1024 + backend: UCX + disable_overlap_scheduler: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.9 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 1024 + backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_tp4_bs256_eplb0_mtp0_con64_ccb-UCX.yaml similarity index 60% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_tp4_bs256_eplb0_mtp0_con64_ccb-UCX.yaml index 3a4b226b8a54..86e443bb700f 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_1k1k_ctx1_tp1_gen1_tp4_bs256_eplb0_mtp0_con64_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: gpt_oss_120b_fp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: GPT-OSS-120B-FP4 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2' + concurrency_list: '64' input_length: 1024 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/gpt_oss_120b-1k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,38 +36,28 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false + print_iter_log: true + max_batch_size: 256 + max_num_tokens: 20000 + tensor_parallel_size: 4 + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 256 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.9 @@ -75,26 +65,29 @@ worker_config: moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 1024 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.9 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 1024 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_dep2_bs512_eplb0_mtp0_con512_ccb-UCX.yaml similarity index 59% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_dep2_bs512_eplb0_mtp0_con512_ccb-UCX.yaml index 6f773ca92606..2e987388cb18 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con2_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_dep2_bs512_eplb0_mtp0_con512_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: gpt_oss_120b_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GPT-OSS-120B-FP4 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2' + concurrency_list: '512' input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/gpt_oss_120b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,62 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false + print_iter_log: true + max_batch_size: 512 + max_num_tokens: 20000 + tensor_parallel_size: 2 + moe_expert_parallel_size: 2 pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: true + attention_dp_config: + batching_wait_iters: 0 + enable_balance: true + timeout_iters: 60 cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 512 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.9 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.9 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con128_ccb-UCX.yaml similarity index 61% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con128_ccb-UCX.yaml index a5b87b237c4a..1e961197d492 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con128_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: gpt_oss_120b_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GPT-OSS-120B-FP4 supported_gpus: - GB200 - GB300 @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '1' + concurrency_list: '128' input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/gpt_oss_120b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,58 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false + print_iter_log: true + max_batch_size: 1024 + max_num_tokens: 20000 + tensor_parallel_size: 4 + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 1024 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.9 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.9 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con4_ccb-UCX.yaml similarity index 62% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con4_ccb-UCX.yaml index 770803ce8ab3..cdaf42d9dc9a 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_8k1k_ctx1_gen3_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1024_eplb0_mtp0_con4_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: gpt_oss_120b_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: GPT-OSS-120B-FP4 supported_gpus: - GB200 - GB300 @@ -24,11 +24,11 @@ benchmark: concurrency_list: '4' input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/gpt_oss_120b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 3 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,58 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false + print_iter_log: true + max_batch_size: 1024 + max_num_tokens: 20000 + tensor_parallel_size: 4 + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 9419 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 1024 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.9 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true - pipeline_parallel_size: 1 print_iter_log: true + max_batch_size: 32 + max_num_tokens: 20000 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.9 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: max_tokens_in_buffer: 8448 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX.yaml similarity index 53% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX.yaml index fa3c3da43612..321a34a99991 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP8_1k1k_ctx1_gen1_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/gpt-oss-120b-fp4_8k1k_ctx1_tp1_gen1_tp4_bs1280_eplb0_mtp0_con1024_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: qwen3_235b_a22b_fp8 - precision: fp8 - model_dir_name: Qwen3-235B-A22B-FP8 + model_name: gpt_oss_120b_fp4 + precision: fp4 + model_dir_name: GPT-OSS-120B-FP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -21,10 +21,10 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '16' - input_length: 1024 + concurrency_list: '1024' + input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/gpt_oss_120b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 @@ -36,7 +36,8 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + OVERRIDE_QUANT_ALGO=W4A8_MXFP4_MXFP8 TRT_LLM_DISABLE_LOAD_WEIGHTS_IN_PARALLEL=True NCCL_GRAPH_REGISTER=0 OMPI_MCA_coll_ucc_enable=0 server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false @@ -44,47 +45,57 @@ accuracy: enable_accuracy_test: false worker_config: gen: + print_iter_log: true tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 2048 - max_seq_len: 2051 + context_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 1280 + max_num_tokens: 20000 + max_seq_len: 11000 + trust_remote_code: true + allreduce_strategy: AUTO + attention_dp_config: + enable_balance: true cuda_graph_config: enable_padding: true - max_batch_size: 128 - print_iter_log: true + max_batch_size: 1280 kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 + enable_block_reuse: false + free_gpu_memory_fraction: 0.85 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 2048 + max_tokens_in_buffer: 9216 backend: UCX - stream_interval: 20 - num_postprocess_workers: 4 - allreduce_strategy: MNNVL disable_overlap_scheduler: false + num_postprocess_workers: 4 + stream_interval: 20 ctx: - max_batch_size: 32 - max_num_tokens: 2048 - max_seq_len: 2051 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: false - pipeline_parallel_size: 1 print_iter_log: true - cuda_graph_config: null - disable_overlap_scheduler: true + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + max_batch_size: 64 + max_num_tokens: 20000 + max_seq_len: 9000 + trust_remote_code: true + cuda_graph_config: + enable_padding: true + max_batch_size: 30 kv_cache_config: - enable_block_reuse: true - free_gpu_memory_fraction: 0.7 + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 2048 + max_tokens_in_buffer: 9216 backend: UCX + disable_overlap_scheduler: true + num_postprocess_workers: 4 + stream_interval: 20 diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml similarity index 57% rename from tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml index 6baf445d017f..1a8af38c8772 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k2_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2-Thinking-NVFP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 8k1k + benchmark_type: 1k1k slurm: script_file: disaggr_torch.slurm partition: @@ -16,18 +16,18 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: gen_only - use_nv_sa_benchmark: false - multi_round: 1 + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '512' - input_length: 8192 + concurrency_list: '2048' + input_length: 1024 output_length: 1024 - dataset_file: datasets/perf-ci/deepseek_r1-8k1k-20480-ratio-1_for_serve.json + dataset_file: datasets/perf-ci/k2_thinking-1k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 - num_ctx_servers: 8 + num_ctx_servers: 1 num_gen_servers: 1 environment: container_mount: @@ -36,76 +36,66 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - enable_layerwise_nvtx_marker: true + print_iter_log: true + max_batch_size: 64 + max_num_tokens: 128 tensor_parallel_size: 32 moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 enable_attention_dp: true enable_lm_head_tp_in_adp: true - pipeline_parallel_size: 1 - max_batch_size: 16 - max_num_tokens: 64 - max_seq_len: 9419 cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 64 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true load_balancer: - num_slots: 288 + num_slots: 384 layer_updates_per_iter: 1 cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + stream_interval: 20 ctx: - enable_layerwise_nvtx_marker: true - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml new file mode 100644 index 000000000000..d8bc75f04e0f --- /dev/null +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml @@ -0,0 +1,98 @@ +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 1k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '4096' + input_length: 1024 + output_length: 1024 + dataset_file: datasets/perf-ci/k2_thinking-1k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 512 + max_num_tokens: 512 + tensor_parallel_size: 8 + moe_expert_parallel_size: 8 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 512 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml similarity index 61% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml index d01e1fac9f7d..eb46f642d4b5 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k2_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2-Thinking-NVFP4 supported_gpus: - GB200 - GB300 @@ -24,11 +24,11 @@ benchmark: concurrency_list: '4' input_length: 1024 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/k2_thinking-1k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,61 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 + print_iter_log: true + max_batch_size: 4 max_num_tokens: 128 - max_seq_len: 2251 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 4 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml similarity index 56% rename from tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml index 07841f3cccef..e7d1a858458b 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k2_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2-Thinking-NVFP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -16,18 +16,18 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: gen_only - use_nv_sa_benchmark: false - multi_round: 1 + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '2048' - input_length: 1024 + concurrency_list: '4096' + input_length: 8192 output_length: 1024 - dataset_file: datasets/perf-ci/deepseek_r1-1k1k-20480-ratio-1_for_serve.json + dataset_file: datasets/perf-ci/k2_thinking-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 - num_ctx_servers: 2 + num_ctx_servers: 1 num_gen_servers: 1 environment: container_mount: @@ -36,76 +36,66 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - enable_layerwise_nvtx_marker: true + print_iter_log: true + max_batch_size: 256 + max_num_tokens: 256 tensor_parallel_size: 16 moe_expert_parallel_size: 16 + pipeline_parallel_size: 1 + context_parallel_size: 1 enable_attention_dp: true enable_lm_head_tp_in_adp: true - pipeline_parallel_size: 1 - max_batch_size: 128 - max_num_tokens: 512 - max_seq_len: 2251 cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 256 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true load_balancer: - num_slots: 288 + num_slots: 384 layer_updates_per_iter: 1 cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + stream_interval: 20 ctx: - enable_layerwise_nvtx_marker: true - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml new file mode 100644 index 000000000000..2229b118f45d --- /dev/null +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml @@ -0,0 +1,107 @@ +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '1024' + input_length: 8192 + output_length: 1024 + dataset_file: datasets/perf-ci/k2_thinking-8k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 128 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 32 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 416 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: Eagle + max_draft_len: 3 + eagle3_one_model: true + speculative_model: Kimi-K2-Thinking-NVFP4-Eagle3 + trust_remote_code: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml new file mode 100644 index 000000000000..97bbef22f102 --- /dev/null +++ b/tests/scripts/perf/disaggregated/kimi-k2-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml @@ -0,0 +1,103 @@ +metadata: + model_name: k2_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2-Thinking-NVFP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '4' + input_length: 8192 + output_length: 1024 + dataset_file: datasets/perf-ci/k2_thinking-8k1k-20480-ratio-1_for_serve.json +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 4 + max_num_tokens: 128 + tensor_parallel_size: 8 + moe_expert_parallel_size: 8 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false + cuda_graph_config: + enable_padding: true + max_batch_size: 4 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + moe_config: + backend: TRTLLM + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: Eagle + max_draft_len: 3 + eagle3_one_model: true + speculative_model: Kimi-K2-Thinking-NVFP4-Eagle3 + trust_remote_code: true + num_postprocess_workers: 4 + stream_interval: 20 + allreduce_strategy: MNNVL + ctx: + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml similarity index 59% rename from tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml index 97f665917e5e..db6fc4827917 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep32_bs64_eplb384_mtp0_con2048_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k25_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2.5-NVFP4 supported_gpus: - GB200 - GB300 @@ -16,15 +16,15 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: gen_only - use_nv_sa_benchmark: false - multi_round: 1 + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '1024' + concurrency_list: '2048' input_length: 1024 output_length: 1024 - dataset_file: datasets/perf-ci/deepseek_r1-1k1k-20480-ratio-1_for_serve.json + dataset_file: hardware: gpus_per_node: 4 num_ctx_servers: 1 @@ -36,70 +36,66 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - enable_layerwise_nvtx_marker: true + print_iter_log: true + max_batch_size: 64 + max_num_tokens: 128 tensor_parallel_size: 32 moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 enable_attention_dp: true enable_lm_head_tp_in_adp: true - pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 32 - max_seq_len: 2251 cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 64 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true load_balancer: - num_slots: 288 + num_slots: 384 layer_updates_per_iter: 1 cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 + stream_interval: 20 ctx: - enable_layerwise_nvtx_marker: true - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml new file mode 100644 index 000000000000..9c70c8bb1ff6 --- /dev/null +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_dep8_bs512_eplb0_mtp0_con4096_ccb-UCX.yaml @@ -0,0 +1,98 @@ +metadata: + model_name: k25_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2.5-NVFP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 1k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '4096' + input_length: 1024 + output_length: 1024 + dataset_file: +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 512 + max_num_tokens: 512 + tensor_parallel_size: 8 + moe_expert_parallel_size: 8 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 512 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml similarity index 63% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml index 1a5bb960bdb7..adc8a13f2e1d 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con4_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_1k1k_ctx1_dep4_gen1_tep4_bs4_eplb0_mtp0_con4_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: qwen3_235b_a22b_fp4 + model_name: k25_thinking_fp4 precision: fp4 - model_dir_name: Qwen3-235B-A22B-FP4 + model_dir_name: Kimi-K2.5-NVFP4 supported_gpus: - GB200 - GB300 @@ -28,7 +28,7 @@ benchmark: hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,61 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false - pipeline_parallel_size: 1 - max_batch_size: 32 + print_iter_log: true + max_batch_size: 4 max_num_tokens: 128 - max_seq_len: 2251 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 4 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 16 + max_num_tokens: 8192 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml similarity index 58% rename from tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml index 0b3c24058ce3..1fbadf14292b 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-r1-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep16_bs256_eplb384_mtp0_con4096_ccb-UCX.yaml @@ -1,7 +1,7 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k25_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2.5-NVFP4 supported_gpus: - GB200 - GB300 @@ -16,18 +16,18 @@ slurm: extra_args: --gres=gpu:4 numa_bind: true benchmark: - mode: gen_only - use_nv_sa_benchmark: false - multi_round: 1 + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '1024' + concurrency_list: '4096' input_length: 8192 output_length: 1024 - dataset_file: datasets/perf-ci/deepseek_r1-8k1k-20480-ratio-1_for_serve.json + dataset_file: hardware: gpus_per_node: 4 - num_ctx_servers: 6 + num_ctx_servers: 1 num_gen_servers: 1 environment: container_mount: @@ -36,70 +36,66 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - enable_layerwise_nvtx_marker: true + print_iter_log: true + max_batch_size: 256 + max_num_tokens: 256 tensor_parallel_size: 16 moe_expert_parallel_size: 16 + pipeline_parallel_size: 1 + context_parallel_size: 1 enable_attention_dp: true enable_lm_head_tp_in_adp: true - pipeline_parallel_size: 1 - max_batch_size: 64 - max_num_tokens: 64 - max_seq_len: 9419 cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 256 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.7 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: - backend: WIDEEP + backend: CUTEDSL + use_low_precision_moe_combine: true load_balancer: - num_slots: 288 + num_slots: 384 layer_updates_per_iter: 1 cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + trust_remote_code: true num_postprocess_workers: 4 + stream_interval: 20 ctx: - enable_layerwise_nvtx_marker: true - max_batch_size: 1 - max_num_tokens: 8448 - max_seq_len: 9419 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.75 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 8448 + max_tokens_in_buffer: 16384 backend: UCX + disable_overlap_scheduler: true + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml new file mode 100644 index 000000000000..2a5319b54ab7 --- /dev/null +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_dep32_bs32_eplb416_mtp3_con1024_ccb-UCX.yaml @@ -0,0 +1,107 @@ +metadata: + model_name: k25_thinking_fp4 + precision: fp4 + model_dir_name: Kimi-K2.5-NVFP4 + supported_gpus: + - GB200 + - GB300 + script_file: disaggr_torch.slurm + benchmark_type: 8k1k +slurm: + script_file: disaggr_torch.slurm + partition: + account: + job_time: 02:00:00 + job_name: unified-benchmark + extra_args: --gres=gpu:4 + numa_bind: true +benchmark: + mode: e2e + use_nv_sa_benchmark: true + multi_round: 8 + benchmark_ratio: 0.8 + streaming: true + concurrency_list: '1024' + input_length: 8192 + output_length: 1024 + dataset_file: +hardware: + gpus_per_node: 4 + num_ctx_servers: 1 + num_gen_servers: 1 +environment: + container_mount: + container_image: + model_path: + trtllm_repo: '' + build_wheel: false + work_dir: + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes + server_env_var: TRTLLM_SERVER_DISABLE_GC=1 +profiling: + nsys_on: false +accuracy: + enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 +worker_config: + gen: + print_iter_log: true + max_batch_size: 32 + max_num_tokens: 128 + tensor_parallel_size: 32 + moe_expert_parallel_size: 32 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: + enable_padding: true + max_batch_size: 32 + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.8 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + load_balancer: + num_slots: 416 + layer_updates_per_iter: 1 + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: Eagle + max_draft_len: 3 + eagle3_one_model: true + speculative_model: Kimi-K2.5-Thinking-Eagle3 + trust_remote_code: true + num_postprocess_workers: 4 + stream_interval: 20 + ctx: + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 + pipeline_parallel_size: 1 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true + cuda_graph_config: null + kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.6 + dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true + cache_transceiver_config: + max_tokens_in_buffer: 16384 + backend: UCX + disable_overlap_scheduler: true + speculative_config: *id001 + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml similarity index 60% rename from tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml index 8cd81db3c117..38047590c1b4 100644 --- a/tests/scripts/perf/disaggregated/deepseek-r1-fp4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp3_con4_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/kimi-k25-thinking-fp4_8k1k_ctx1_dep4_gen1_tep8_bs4_eplb0_mtp3_con4_ccb-UCX.yaml @@ -1,12 +1,12 @@ metadata: - model_name: deepseek_r1_0528_fp4_v2 + model_name: k25_thinking_fp4 precision: fp4 - model_dir_name: DeepSeek-R1-0528-FP4-v2 + model_dir_name: Kimi-K2.5-NVFP4 supported_gpus: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -22,13 +22,13 @@ benchmark: benchmark_ratio: 0.8 streaming: true concurrency_list: '4' - input_length: 1024 + input_length: 8192 output_length: 1024 dataset_file: hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,71 +36,68 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 4 + max_num_tokens: 128 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 4 kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.9 + free_gpu_memory_fraction: 0.8 dtype: fp8 moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true + speculative_config: &id001 + decoding_type: Eagle + max_draft_len: 3 + eagle3_one_model: true + speculative_model: Kimi-K2.5-Thinking-Eagle3 + trust_remote_code: true num_postprocess_workers: 4 - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + stream_interval: 20 allreduce_strategy: MNNVL ctx: - max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 + print_iter_log: true + max_batch_size: 2 + max_num_tokens: 8768 tensor_parallel_size: 4 moe_expert_parallel_size: 4 - enable_attention_dp: true pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 16384 backend: UCX - speculative_config: - decoding_type: MTP - num_nextn_predict_layers: 3 + disable_overlap_scheduler: true + speculative_config: *id001 + trust_remote_code: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_dep8_bs128_eplb0_mtp0_con1024_ccb-UCX.yaml similarity index 62% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_dep8_bs128_eplb0_mtp0_con1024_ccb-UCX.yaml index 6cd7320e367b..2c61fcb2c68a 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con1_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_dep8_bs128_eplb0_mtp0_con1024_ccb-UCX.yaml @@ -6,7 +6,7 @@ metadata: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '1' - input_length: 1024 + concurrency_list: '1024' + input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/qwen3_235b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,65 +36,60 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: + print_iter_log: true + max_batch_size: 128 + max_num_tokens: 128 tensor_parallel_size: 8 moe_expert_parallel_size: 8 - enable_attention_dp: false pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: true + enable_lm_head_tp_in_adp: true cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 128 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.9 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32768 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: + print_iter_log: true max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true + max_num_tokens: 32768 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: TRTLLM + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32768 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml b/tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_tep4_bs64_eplb0_mtp0_con64_ccb-UCX.yaml similarity index 62% rename from tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml rename to tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_tep4_bs64_eplb0_mtp0_con64_ccb-UCX.yaml index 41598d88f684..b3bdfd571d97 100644 --- a/tests/scripts/perf/disaggregated/Qwen3-235B-A22B-FP4_1k1k_ctx1_gen4_tep8_bs32_eplb0_mtp0_con16_ccb-UCX.yaml +++ b/tests/scripts/perf/disaggregated/qwen3-235b-fp4_8k1k_ctx1_tp1_gen1_tep4_bs64_eplb0_mtp0_con64_ccb-UCX.yaml @@ -6,7 +6,7 @@ metadata: - GB200 - GB300 script_file: disaggr_torch.slurm - benchmark_type: 1k1k + benchmark_type: 8k1k slurm: script_file: disaggr_torch.slurm partition: @@ -21,14 +21,14 @@ benchmark: multi_round: 8 benchmark_ratio: 0.8 streaming: true - concurrency_list: '16' - input_length: 1024 + concurrency_list: '64' + input_length: 8192 output_length: 1024 - dataset_file: + dataset_file: datasets/perf-ci/qwen3_235b-8k1k-20480-ratio-1_for_serve.json hardware: gpus_per_node: 4 num_ctx_servers: 1 - num_gen_servers: 4 + num_gen_servers: 1 environment: container_mount: container_image: @@ -36,38 +36,28 @@ environment: trtllm_repo: '' build_wheel: false work_dir: - worker_env_var: "TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 MIMALLOC_PURGE_DELAY=0 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes" + worker_env_var: TLLM_LOG_LEVEL=INFO TRTLLM_SERVER_DISABLE_GC=1 TRTLLM_WORKER_DISABLE_GC=1 TRTLLM_ENABLE_PDL=1 ENROOT_ALLOW_DEV=yes server_env_var: TRTLLM_SERVER_DISABLE_GC=1 profiling: nsys_on: false accuracy: enable_accuracy_test: false + model: local-completions + tasks: gsm8k + model_args_extra: num_concurrent=512,max_retries=3,tokenized_requests=false,timeout=1200,max_gen_toks=256,max_length=4096 worker_config: gen: - tensor_parallel_size: 8 - moe_expert_parallel_size: 8 - enable_attention_dp: false + print_iter_log: true + max_batch_size: 64 + max_num_tokens: 64 + tensor_parallel_size: 4 + moe_expert_parallel_size: 4 pipeline_parallel_size: 1 - max_batch_size: 32 - max_num_tokens: 128 - max_seq_len: 2251 + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: enable_padding: true - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 32 - - 64 - - 128 - - 256 - - 512 - - 768 - - 1024 - - 2048 - print_iter_log: true + max_batch_size: 64 kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.9 @@ -75,26 +65,29 @@ worker_config: moe_config: backend: TRTLLM cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32768 backend: UCX - stream_interval: 20 + disable_overlap_scheduler: true num_postprocess_workers: 4 - allreduce_strategy: MNNVL + stream_interval: 20 ctx: + print_iter_log: true max_batch_size: 4 - max_num_tokens: 4608 - max_seq_len: 2251 - tensor_parallel_size: 4 - moe_expert_parallel_size: 4 - enable_attention_dp: true + max_num_tokens: 32768 + tensor_parallel_size: 1 + moe_expert_parallel_size: 1 pipeline_parallel_size: 1 - print_iter_log: true + context_parallel_size: 1 + enable_attention_dp: false cuda_graph_config: null - disable_overlap_scheduler: true kv_cache_config: enable_block_reuse: false - free_gpu_memory_fraction: 0.85 + free_gpu_memory_fraction: 0.6 dtype: fp8 + moe_config: + backend: TRTLLM + use_low_precision_moe_combine: true cache_transceiver_config: - max_tokens_in_buffer: 4608 + max_tokens_in_buffer: 32768 backend: UCX + disable_overlap_scheduler: true diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml index c12ae7ab9f6e..a79d99deb2d4 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx1_gen1_dep32_bs32_eplb288_mtp0_con1024_ccb-NIXL.yaml @@ -103,7 +103,7 @@ worker_config: cuda_graph_config: null disable_overlap_scheduler: true moe_config: - backend: TRTLLM + backend: CUTEDSL kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.85 diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml index c68474bc9529..1c4e696fbd96 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep16_bs128_eplb288_mtp3_con2048_ccb-NIXL.yaml @@ -106,7 +106,7 @@ worker_config: cuda_graph_config: null disable_overlap_scheduler: true moe_config: - backend: TRTLLM + backend: CUTEDSL kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.85 diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml index 263768adc82d..7cd38afce6f6 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_1k1k_ctx2_gen1_dep48_bs16_eplb288_mtp3_con12288_ccb-DEFAULT.yaml @@ -102,7 +102,7 @@ worker_config: cuda_graph_config: null disable_overlap_scheduler: true moe_config: - backend: TRTLLM + backend: CUTEDSL kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.85 diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml index 2df63073a052..3c5621d7b662 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx2_gen1_dep32_bs128_eplb288_mtp3_con1024_ccb-DEFAULT.yaml @@ -111,7 +111,7 @@ worker_config: free_gpu_memory_fraction: 0.75 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 8448 backend: DEFAULT diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml index 9a899579cacc..b498b3e7d1cf 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx6_gen1_dep16_bs64_eplb288_mtp0_con1024_ccb-NIXL.yaml @@ -107,7 +107,7 @@ worker_config: free_gpu_memory_fraction: 0.75 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 8448 backend: NIXL diff --git a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml index 8b558ba9c902..a791061049b7 100644 --- a/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml +++ b/tests/scripts/perf/disaggregated/wideep_deepseek-v32-fp4_8k1k_ctx8_gen1_dep32_bs16_eplb288_mtp3_con512_ccb-NIXL.yaml @@ -110,7 +110,7 @@ worker_config: free_gpu_memory_fraction: 0.75 dtype: fp8 moe_config: - backend: TRTLLM + backend: CUTEDSL cache_transceiver_config: max_tokens_in_buffer: 8448 backend: NIXL