[NVIDIA][AgentX] Move GB300 DSV4.1-Flash Dynamo + SGLang to the bonus-fixed nightlies with prefill-HiCache disagg points and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang 升级到含 bonus 修复的 nightly，分离式改用 prefill HiCache，TP4 c1 启用 megakernel

inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-dep16-hicache-variants.yaml

@@ -0,0 +1,247 @@
+schema: 2
+
+base:
+  name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-dep16-hicache
+  identity:
+    model: {}
+    frameworks:
+      dynamo: "1.7.0.dev20261026"
+      sglang: "0.0.0.dev1+g5884e4fe"
+  dynamo:
+    install: false
+    request_plane: tcp
+    event_plane: zmq
+  # gb300-nv serves this model from the Hub; read it from the shared mounted cache.
+  environment:
+    HF_HUB_CACHE: /hf_hub_cache
+  # Tachometer's per-node exporters slow decode steps and add head-node host memory;
+  # InferenceX does not read its output.
+  observability:
+    tachometer:
+      enabled: false
+  slurm:
+    time_limit: "6:00:00"
+  health_check:
+    max_attempts: 1440
+    interval_seconds: 10
+  resources:
+    gpu_type: gb300
+    gpus_per_node: 4
+  services:
+    - name: etcd
+      type: etcd
+      placement:
+        node: infra
+    # Disables Dynamo's per-request lifecycle span export; InferenceX does not read it.
+    - name: otel-collector
+      type: otel-collector
+      enabled: false
+  frontend:
+    type: dynamo
+    enable_multiple_frontends: false
+    numa_bind: true
+    # One CPU node hosts the frontend, etcd and the AgentX client.
+    placement:
+      node: dedicated
+    env:
+      DYN_ROUTER_TEMPERATURE: "0"
+      DYN_ROUTER_QUEUE_THRESHOLD: None
+      DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600"
+      DYN_TOKENIZER: "fastokens"
+      DYN_TOKENIZER_CACHE: "1"
+      DYN_TOKENIZER_CACHE_BYTES: "8000000000"
+      DYN_TCP_CHANNEL_BUFFER: "128"
+      DYN_TCP_REQUEST_TIMEOUT: "60"
+    args:
+      router-mode: kv
+      router-session-affinity-ttl-secs: "14400"
+      active-decode-blocks-threshold: "None"
+      active-prefill-tokens-threshold: "None"
+      active-prefill-tokens-threshold-frac: "None"
+  engine: sglang
+  roles:
+    prefill:
+      nodes: 5
+      gpus: 2
+      env:
+        PYTHONUNBUFFERED: "1"
+        SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE: "1"
+        SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT: per_rank
+        SGLANG_DSV4_KV_LAYOUT: v41_new
+        SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600"
+        SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600"
+        SGLANG_ENABLE_PREFILL_WAR_READ_DONE: "1"
+        SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True"
+        SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "65536"
+        SGLANG_OPT_DSV41_BF16_INDEX_LOGITS: "1"
+        SGLANG_OPT_DSV41_MEGA_MHC_PREFILL: "1"
+        NCCL_MNNVL_ENABLE: "1"
+        NCCL_CUMEM_ENABLE: "1"
+        MC_FORCE_MNNVL: "1"
+        MOONCAKE_PROTOCOL: rdma
+        MOONCAKE_DEVICE: "@fabric.rdma-devices"
+        MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
+        # Some GB300 nodes list a BMC interface with the same address on every node first.
+        MC_TCP_BIND_ADDRESS: "{node}"
+        WITH_NVIDIA_PEERMEM: "0"
+        # Dynamo 1.7 backend admission rejects turns with 529 once a DP rank is full;
+        # the prefill scheduler queues them instead.
+        DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "10000"
+        DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "10000"
+        DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT: "10000"
+      args:
+        served-model-name: deepseek-ai/DeepSeek-V4.1-Flash
+        trust-remote-code: true
+        tensor-parallel-size: 2
+        tokenizer-backend: fastokens
+        router-mode: kv
+        router-session-affinity-ttl-secs: "14400"
+        data-parallel-size: 2
+        expert-parallel-size: 2
+        enable-dp-attention: true
+        enable-dp-lm-head: true
+        enable-dp-attention-local-control-broadcast: true
+        moe-a2a-backend: megamoe
+        dsv4-attn-backend: flashmla_mega
+        disaggregation-transfer-backend: mooncake
+        chunked-prefill-size: 65536
+        max-prefill-tokens: 32768
+        # 64 per DP rank, as vLLM's prefill max-num-seqs; absorbs the client's turn burst.
+        max-running-requests: 128
+        swa-prefix-tails: 2048
+        enable-decoder-swa-bounded-replay: true
+        enable-encoder-swa-bounded-replay: true
+        # Local host-DRAM tier: device-evicted prefixes reload from host memory off the
+        # forward path; an external L3 lookup blocks each layer of the forward.
+        enable-hierarchical-cache: true
+        hicache-ratio: 1.8
+        hicache-mem-layout: page_first_direct
+        hicache-io-backend: direct
+        hicache-write-policy: write_back
+        mem-fraction-static: 0.85
+        speculative-algorithm: DSPARK
+        speculative-dspark-block-size: 5
+        speculative-num-steps: 5
+        reasoning-parser: auto
+        tool-call-parser: auto
+        watchdog-timeout: 3600
+        enable-metrics: true
+        weight-loader-prefetch-checkpoints: true
+        weight-loader-drop-cache-after-load: false
+        model-loader-extra-config: '{"enable_multithread_load":true}'
+      kv_events: true
+    decode:
+      nodes: 4
+      workers: 1
+      gpus: 16
+      env:
+        PYTHONUNBUFFERED: "1"
+        # Decode ranks read Engram rows from peer shards instead of a host table.
+        SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS: "1"
+        SGLANG_DSV4_KV_LAYOUT: v41_new
+        # One gateway process saturates a CPU core serving 16 DP ranks; with 8, one
+        # instance stalls at high concurrency.
+        DYN_SGL_GATEWAY_WORKERS: "16"
+        SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600"
+        SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600"
+        SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True"
+        SGLANG_OPT_DEEPGEMM_MEGA_GATE: "1"
+        SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384"
+        SGLANG_OPT_DSV41_BF16_INDEX_LOGITS: "1"
+        SGLANG_OPT_DSV41_MEGA_MHC_PREFILL: "1"
+        NCCL_MNNVL_ENABLE: "1"
+        NCCL_CUMEM_ENABLE: "1"
+        MC_FORCE_MNNVL: "1"
+        MOONCAKE_PROTOCOL: rdma
+        MOONCAKE_DEVICE: "@fabric.rdma-devices"
+        MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
+        MC_TCP_BIND_ADDRESS: "{node}"
+        WITH_NVIDIA_PEERMEM: "0"
+        DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "10000"
+        DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "10000"
+        DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT: "10000"
+      args:
+        served-model-name: deepseek-ai/DeepSeek-V4.1-Flash
+        trust-remote-code: true
+        tensor-parallel-size: 16
+        tokenizer-backend: fastokens
+        router-mode: round-robin
+        load-balance-method: total_tokens
+        data-parallel-size: 16
+        expert-parallel-size: 16
+        enable-dp-attention: true
+        enable-dp-lm-head: true
+        enable-dp-attention-local-control-broadcast: true
+        enable-fp32-lm-head: true
+        moe-a2a-backend: megamoe
+        dsv4-attn-backend: flashmla_mega
+        disaggregation-transfer-backend: mooncake
+        disaggregation-decode-extra-slots: 256
+        stream-interval: 32
+        swa-prefix-tails: 2048
+        enable-decoder-swa-bounded-replay: true
+        mem-fraction-static: 0.85
+        enable-eplb: true
+        ep-num-redundant-experts: 16
+        eplb-rebalance-num-iterations: 15000
+        expert-distribution-recorder-buffer-size: 1000
+        speculative-algorithm: DSPARK
+        speculative-dspark-block-size: 5
+        speculative-num-steps: 5
+        reasoning-parser: auto
+        tool-call-parser: auto
+        watchdog-timeout: 3600
+        enable-metrics: true
+        weight-loader-prefetch-checkpoints: true
+        weight-loader-drop-cache-after-load: false
+        model-loader-extra-config: '{"enable_multithread_load":true}'
+      kv_events: false
+  sbatch_directives:
+    mem: "0"
+    exclusive: ""
+  srun_options:
+    container-remap-root: ""
+    # Engram host tables need the node's full memory in every worker step.
+    mem: "0"
+  benchmark:
+    placement:
+      node: dedicated
+    colocate_with_frontend: true
+    env:
+      AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
+      AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
+      AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:"
+      # The client splits this value on whitespace, so it has none.
+      AIPERF_EXTRA_INPUTS: '{"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}'
+      AIPERF_DATASET_MMAP_CACHE_ENABLED: "true"
+
+override_9p1d_dep2_dep16_c7480:
+  name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-dep16-hicache-9p1d-dep2-dep16-c7480
+  roles:
+    prefill:
+      # Nine DEP2 workers leave two GPUs of the fifth node idle.
+      workers: 9
+    decode:
+      args:
+        max-running-requests: 2048
+        cuda-graph-max-bs-decode: 128
+
+override_10p1d_dep2_dep16_c7480:
+  name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-dep16-hicache-10p1d-dep2-dep16-c7480
+  roles:
+    prefill:
+      workers: 10
+    decode:
+      args:
+        max-running-requests: 2048
+        cuda-graph-max-bs-decode: 128
+
+override_10p1d_dep2_dep16_b160_c7480:
+  name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-dep16-hicache-10p1d-dep2-dep16-b160-c7480
+  roles:
+    prefill:
+      workers: 10
+    decode:
+      args:
+        max-running-requests: 2560
+        cuda-graph-max-bs-decode: 160

inferencex-e2e/configs/nvidia-master.yaml

@@ -9718,6 +9718,68 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg:
           ep: 16
           dp-attn: true
 
+dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache:
+  image: lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe@sha256:61b9083bdcb7cf3741366f487cb996c05ea33ad2ddddc820d10cbc1c0cfb0a86
+  model: deepseek-ai/DeepSeek-V4.1-Flash
+  model-prefix: dsv41flash
+  runner: cluster:gb300-nv
+  precision: fp4
+  framework: dynamo-sglang
+  router: { name: dynamo-router, version: "1.7.0.dev20261026" }
+  kv-p2p-transfer: mooncake
+  multinode: true
+  disagg: true
+  srt-recipe-dir: dsv41flash/sglang/gb300-fp4/agentx
+  scenarios:
+    agentic-coding:
+    - dram-utilization: 0.80
+      search-space:
+      - spec-decoding: mtp
+        conc-list: [7480]
+        srt-recipe: disagg-dep16-hicache-variants.yaml:override_9p1d_dep2_dep16_c7480
+        kv-offloading: dram
+        kv-offload-backend: { name: hicache }
+        prefill:
+          num-worker: 9
+          tp: 2
+          ep: 2
+          dp-attn: true
+        decode:
+          num-worker: 1
+          tp: 16
+          ep: 16
+          dp-attn: true
+      - spec-decoding: mtp
+        conc-list: [7480]
+        srt-recipe: disagg-dep16-hicache-variants.yaml:override_10p1d_dep2_dep16_c7480
+        kv-offloading: dram
+        kv-offload-backend: { name: hicache }
+        prefill:
+          num-worker: 10
+          tp: 2
+          ep: 2
+          dp-attn: true
+        decode:
+          num-worker: 1
+          tp: 16
+          ep: 16
+          dp-attn: true
+      - spec-decoding: mtp
+        conc-list: [7480]
+        srt-recipe: disagg-dep16-hicache-variants.yaml:override_10p1d_dep2_dep16_b160_c7480
+        kv-offloading: dram
+        kv-offload-backend: { name: hicache }
+        prefill:
+          num-worker: 10
+          tp: 2
+          ep: 2
+          dp-attn: true
+        decode:
+          num-worker: 1
+          tp: 16
+          ep: 16
+          dp-attn: true
+
 dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg:
   image: vllm/vllm-openai:cu134-nightly-aarch64-dev-20261019
   model: deepseek-ai/DeepSeek-V4.1-Flash

inferencex-e2e/perf-changelog.yaml

@@ -11404,3 +11404,15 @@
     - "TP2 c16-c64, the unsearched TP4 c4-c64 recipe variants, the DSpark speculative config (5 drafts, probabilistic drafting) and mono decode are unchanged."
     - "Draft data types unchanged: the DSpark draft (mtp.0-2 in the target checkpoint) loads with the checkpoint's FP8 config, as shipped (FP8 E4M3 with 32x32 UE8M0 scales for attention, shared experts and main_proj, MXFP4 routed experts, BF16 gate, norms and heads), inherits the target fp8_ds_mla KV cache, and the mono draft kernels read those weights in place."
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4480
+
+- config-keys:
+    - dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg
+    - dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg
+    - dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache
+  scenario-type: [agentic-coding]
+  description:
+    - "Add dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache on nightly-dev-cu13-20261027-5884e4fe (Dynamo 1.7.0.dev20261026), pinned by digest: DEP2 prefill and one DEP16 decode at c7480 with 9 prefill workers (decode CUDA graph batch 128), 10 prefill workers (batch 128) and 10 prefill workers (batch 160, max-running-requests 2560)."
+    - "Prefill keeps evicted prefixes in a local host-DRAM HiCache tier (ratio 1.8, write-back, direct I/O, mem-fraction-static 0.85) instead of the Mooncake external linker, whose layer-wise L3 reads block the forward pass and stall prefill after the client's turn burst at c7480. Decode admits transferred requests every step (SGLang's default disaggregation-decode-polling-interval of 1; the dspark-disagg key sets 8)."
+    - "Dynamo 1.7 backend admission limits are raised to 10000 so waiting turns queue in the prefill scheduler instead of failing with 529, and runners/srt-slurm/patches backports NVIDIA/srt-slurm#629 so the readiness wait asks /health?instances=true when /health is status-only."
+    - "The agg and existing dspark-disagg keys are unchanged; they rerun in the same sweep because AgentX plots all dynamo-sglang points of this model and GPU as one curve."
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

inferencex-e2e/runners/srt-slurm/patches/629-dynamo-health-instances.patch

@@ -0,0 +1,48 @@
+diff --git a/src/srtctl/frontends/dynamo.py b/src/srtctl/frontends/dynamo.py
+--- a/src/srtctl/frontends/dynamo.py
++++ b/src/srtctl/frontends/dynamo.py
+@@ -12,6 +12,7 @@ import shlex
+ import threading
+ from typing import TYPE_CHECKING, Any, ClassVar, Literal, cast
+
++import requests
+ import yaml
+
+ from srtctl.backends.vllm import VLLMBackend
+@@ -42,6 +43,8 @@ logger = logging.getLogger(__name__)
+
+ ROUTER_POLICY_CONFIG_FILENAME = "router_policy_config.yaml"
+ ROUTER_POLICY_CONFIG_CONTAINER_PATH = f"/logs/{ROUTER_POLICY_CONFIG_FILENAME}"
++# Since ai-dynamo/dynamo#16454 (a789babbe3) /health is status-only unless asked for its instance list.
++HEALTH_INSTANCES_ENDPOINT = "/health?instances=true"
+
+
+ def _vllm_mode_config(backend: VLLMBackend, mode: str) -> dict[str, Any]:
+@@ -176,9 +179,26 @@ class DynamoFrontend(DynamicFrontend):
+         expected_prefill: int,
+         expected_decode: int,
+     ) -> WorkerHealthResult:
+-        """Parse dynamo /health endpoint response."""
++        """Count the generate instances in a Dynamo /health instance listing (see ``probe_ready``)."""
+         return check_dynamo_health(response_json, expected_prefill, expected_decode)
+
++    def probe_ready(
++        self, host: str, port: int, expected_prefill: int, expected_decode: int, config: Any
++    ) -> WorkerHealthResult:
++        """Count instances from ``/health``, asking ``/health?instances=true`` when it answers status-only.
++
++        Frontends before ai-dynamo/dynamo#16454 always list instances in ``/health`` and get the one
++        request they always got; newer ones answer ``{"status": "healthy", "stage": "ready"}`` there.
++        """
++        for endpoint in (self.health_endpoint, HEALTH_INSTANCES_ENDPOINT):
++            response = requests.get(f"http://{host}:{port}{endpoint}", timeout=5.0)
++            if response.status_code != 200:
++                return WorkerHealthResult(ready=False, message=f"{endpoint} returned HTTP {response.status_code}")
++            body = response.json()
++            if "instances" in body:
++                break
++        return self.parse_health(body, expected_prefill, expected_decode)
++
+     def health_expectations(self, config: Any, processes: list["Process"] | None) -> tuple[int, int, str]:
+         """Dynamo's /health counts registered generate instances.
+

inferencex-e2e/runners/srt-slurm/patches/README.md

@@ -9,3 +9,4 @@ Each patch backports an upstream fix that the shared submodule pin does not incl
 | Patch | Upstream PR | Fix |
 |-------|-------------|-----|
 | `573-participating-gpus.patch` | [NVIDIA/srt-slurm#573](https://github.com/NVIDIA/srt-slurm/pull/573) | Record only the GPUs workers occupy on a worker node; an exclusive node's exporter (DCGM or AMD) reports every GPU, which the collector otherwise flags as `unexpected_device`. The pin includes #573, but upstream merged a different design: it keeps every row and excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. [`infx/results/power/multinode.py`](../../../infx/results/power/multinode.py) does not apply that excusal yet, so keep this filter until it does. |
+| `629-dynamo-health-instances.patch` | [NVIDIA/srt-slurm#629](https://github.com/NVIDIA/srt-slurm/pull/629) | Dynamo frontends since ai-dynamo/dynamo#16454 (1.7 dev builds) answer `/health` status-only and list instances only for `/health?instances=true`, so the readiness wait never counted any workers. `DynamoFrontend.probe_ready` reads `/health` and asks `/health?instances=true` only when the answer has no `instances` key; older frontends get the same single request. |