[None][test] Add AgentX DeepSeek-V4-Pro-DSpark perf-sanity lanes on GB300

jenkins/L0_Test.groovy

@@ -6557,6 +6557,19 @@ def launchTestJobs(pipeline, testFilter, globalVars)
     // Nemotron-Ultra-V3 con9832 (8k64k) and con1197 (50k2k) are ctx_only-only:
     // their full 68-/72-GPU e2e+gen_only disagg topologies are intentionally not
     // created; the ctx_only ids run in the 4-GPU multi_gpus post-merge stage.
+    // GB300 DeepSeek-V4-Pro-DSpark, AgentX agentic trace replay.
+    // These lanes replay a ~1M-token multi-turn conversation trace for a fixed
+    // wall-clock duration instead of a fixed prompt count, so they are pinned to
+    // aws-cmh where the DSpark checkpoint and the trace corpus are staged.
+    // 6 Nodes: ctx2 (2 nodes, 8 GPUs each) + gen1 (2 nodes, 8 GPUs) = 24 GPUs
+    multiNodesSBSAConfigs += buildStageConfigs(
+        "GB300-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-AgentX-CTX2-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge",
+        "gb300-flex-aws-cmh",
+        "l0_gb300_multi_nodes_perf_sanity_ctx2_node2_gpu8_gen1_node2_gpu8",
+        1,
+        24,
+        6
+    )
     multiNodesSBSAConfigs = cbtsResizeSplits(multiNodesSBSAConfigs)
     fullSet += multiNodesSBSAConfigs.keySet()
 

jenkins/scripts/perf/local/README.md

@@ -99,3 +99,106 @@ python3 submit.py --test-list "perf/test_perf_sanity.py::test_e2e[disagg-e2e-gb2
     --mounts $mounts \
     --llm-models-root $llm_models_path
 ```
+
+---
+
+## Running the AgentX perf-sanity lane
+
+The AgentX client replays a recorded multi-turn conversation corpus for a fixed
+wall-clock window (`AGENTX_DURATION`, default 3600 s) rather than a fixed number of
+fixed-shape prompts, so `isl`/`osl`/`iterations` in the config are descriptive and
+the run is judged on duration coverage, not a request count. Example test id
+(`_upload` is stripped for local runs, so nothing reaches OpenSearch):
+
+```
+perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-v4-pro-dspark_agentx_con1156_ctx2_dep8_gen1_dep8_eplb0_dspark3_ccb-NIXL]
+```
+
+### Environment variables
+
+`submit.py` reads `EXTRA_CONTAINER_EXPORTS` -- a `;`-separated `KEY=VALUE` list --
+at **generation time** and splices it into the four per-role env prefixes
+(`CTX_WORKER_ENV_VARS`, `GEN_WORKER_ENV_VARS`, `SERVER_ENV_VARS`,
+`BENCHMARK_ENV_VARS`). Export it before running `submit.py`, not at `sbatch` time:
+
+```bash
+export EXTRA_CONTAINER_EXPORTS="HF_HOME=$hf_cache;HF_HUB_CACHE=$hf_cache/hub;HF_DATASETS_CACHE=$hf_cache/datasets;PIP_CACHE_DIR=$work_dir/.pip-cache"
+```
+
+Set all three HF variables, not just `HF_HOME`: `submit.py` appends its own
+`HF_HOME=/tmp/hf_home` **after** the splice for the ctx and gen worker roles, and
+the later assignment wins. `HF_HUB_CACHE` and `HF_DATASETS_CACHE` are never
+overridden and outrank `HF_HOME` in `huggingface_hub` / `datasets`, making the
+result independent of splice order.
+
+`dataset_file` in the config is an aiperf `--public-dataset` loader name, not a
+path, and is fetched from Hugging Face at runtime. Either warm the cache above
+before submitting, or confirm the compute nodes can reach the HF CDN.
+
+The `AGENTX_*` knobs are not `submit.py` flags -- they come from `client_env_var`
+in the config YAML, so exporting them in your shell has no effect. Edit the YAML
+to change one. `AGENTX_DURATION` is the main cost knob; keep `AGENTX_SEED`
+(default 42) fixed when comparing runs.
+
+```yaml
+client_env_var: 'AGENTX_MAX_CTX=996579 AGENTX_DURATION=3600 AGENTX_WARMUP_PER_LANE=3'
+```
+
+### Generate and submit
+
+```bash
+python3 submit.py \
+    --test-list "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-v4-pro-dspark_agentx_con1156_ctx2_dep8_gen1_dep8_eplb0_dspark3_ccb-NIXL]" \
+    --draft-launch-sh $trtllm/jenkins/scripts/perf/disaggregated/slurm_launch_draft.sh \
+    --launch-sh $work_dir/slurm_launch.sh \
+    --install-sh $trtllm/jenkins/scripts/perf/local/slurm_install.sh \
+    --run-sh $trtllm/jenkins/scripts/perf/local/slurm_run.sh \
+    --llm-src $trtllm \
+    --work-dir $work_dir \
+    --llm-models-root $llm_models_path \
+    --partition $partition \
+    --account $account \
+    --job-name agentx_test \
+    --image $image \
+    --mounts $mounts \
+    --install-mode wheel \
+    --wheel-path $wheel \
+    --cluster-name $cluster
+
+cd $work_dir && sbatch slurm_launch.sh
+```
+
+`--cluster-name` selects the UCX and env rules in `cluster_env.py`; an unmatched
+(cluster, GPU) pair falls through to a catch-all that pins no transport, changing
+performance silently. Use a fresh `--work-dir` every run -- a reused disaggregated
+work directory reads stale hostname files and hangs. This lane takes 6 nodes /
+24 GPUs and roughly 2.5 h including the wheel install, so allow a 4 h limit.
+
+### Viewing the perf results
+
+Artifacts land under `$work_dir/<case-name>/`:
+
+| Path | Contents |
+|---|---|
+| `agentx.0.0/concurrency_1156/profile_export_aiperf.json` | metrics, percentiles, `submission_valid`, duration coverage (client log alongside in `logs/aiperf.log`) |
+| `trtllm-benchmark.0.0.log` | human-readable metric block |
+| `{ctx,gen}_server_*.log`, `disagg_server.log` | server logs |
+
+Gate on the export rather than on the pytest exit status:
+
+```bash
+python3 -c "
+import json
+d = json.load(open('$work_dir/<case>/agentx.0.0/concurrency_1156/profile_export_aiperf.json'))
+m = d['metadata']
+print('submission_valid', m.get('submission_valid'))
+print('was_cancelled   ', m.get('was_cancelled'))
+print('errors          ', m.get('error_summary'))
+for p in m.get('metric_duration_coverage') or []:
+    print('coverage', p)
+"
+```
+
+Coverage reports the TTFT and ITL sample ratios against the requested window, so a
+run that only partly covered it is detectable instead of quietly averaging a
+truncated segment. A green pytest summary alone is not a passing stage.

tests/integration/defs/.test_durations

@@ -930,6 +930,7 @@
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_qwen3-235b-fp4_8k1k_con1024_ctx1_tp1_gen1_dep8_eplb0_mtp0_ccb-NIXL]": 2103.162375,
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL]": 7666.37925,
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-r1-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL]": 2542.9724666666666,
+   "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_deepseek-v4-pro-dspark_agentx_con1156_ctx2_dep8_gen1_dep8_eplb0_dspark3_ccb-NIXL]": 5400.0,
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL]": 3729.2565,
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_nemotron-ultra-v3-fp4_50k2k_con12_ctx1_dep4_gen6_tep4_eplb0_mtp6_ccb-NIXL]": 1731.7142,
    "perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb300_nemotron-ultra-v3-fp4_50k2k_con178_ctx5_dep4_gen1_dep4_eplb0_mtp6_ccb-NIXL]": 1023.3791666666666,

tests/integration/defs/perf/README_test_perf_sanity.md

@@ -117,6 +117,39 @@ pass. `process_and_upload_test_results` therefore looks history up against a
 baseline branch — `PERF_BASELINE_BRANCH`, default `main` — using a lookup-only copy
 of the data. The uploaded documents keep their true `s_branch`.
 
+Case identity is keyed on `s_test_case_name` (plus GPU type, runtime and branch), and a
+disaggregated case name embeds its config stem — so an agentx lane, whose stem contains
+`agentx`, already forms its own baseline population without needing an extra match key.
+`s_benchmark_client` is still uploaded, as a reportable record of which load generator
+drove the lane, but it does not participate in matching. The built-in client uploads `""`
+(not `"default"`) so the column reads consistently against records written before the
+field existed.
+
+## Benchmark Clients
+
+A disaggregated `benchmark_config` may select which load generator drives the lane via
+the optional `benchmark_client` key. Any value other than the two below is rejected at
+config-parse time rather than silently falling through to the default client.
+
+| `benchmark_client` | Client | Notes |
+|---|---|---|
+| *(omitted / `""`)* | `tensorrt_llm/serve/scripts/benchmark_serving.py` | Default. Fixed request count from `dataset_file` + `concurrency_list`. |
+| `agentx` | `perf/agentx_client.py` | AgentX agentic multi-turn trace replay (a fork of `aiperf`). |
+
+**AgentX specifics**:
+- Duration-bounded, not count-bounded: the run replays a conversation trace for
+  `AGENTX_DURATION` seconds, so `dataset_file` names an `aiperf` dataset loader (fetched
+  from HF), *not* a path — `get_dataset_dir` must not be applied to it.
+- `concurrency_list` is the whole-cluster total and is passed through un-multiplied.
+- Tunables are passed through `client_env_var` (`AGENTX_MAX_CTX`, `AGENTX_DURATION`,
+  `AGENTX_WARMUP_PER_LANE`). Artifacts land in
+  `{test_output_dir}/agentx.{server_idx}.{client_idx}/concurrency_{N}/`.
+- `al` (Mean Avg Decoded Tokens per Iter) is exempt from the spec-decoding hard-fail for
+  agentx lanes: it derives from a TRT-LLM-specific per-response field that `aiperf` does
+  not propagate. This loses no signal as long as the lane pins the accepted length with
+  `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS` (set per-yaml), which makes `al` a
+  restatement of a configured constant rather than a measurement.
+
 ## Overview
 
 - Run performance sanity benchmarks across multiple model configs
@@ -191,6 +224,10 @@ There are two modes for perf sanity tests: aggregated (aggr) and disaggregated (
 
 **Use Case**: Disaggregated architecture where model runs across multiple nodes with separate context (prefill) and generation (decode) servers.
 
+**Optional `benchmark_config` keys**: `benchmark_client` selects a non-default load
+generator (see [Benchmark Clients](#benchmark-clients)); `client_env_var` passes
+client-side environment variables through to it.
+
 ## Test Case Formats
 
 In each test db yml file (with keyword `perf_sanity`), there are four test types: