[DO NOT MERGE][debug] Diffusion CI perf: controlled A/B experiments on CI H100

[niehen6174]

Do not merge or review. Temporary debug PR; I will close it as soon as the run finishes.

Why

Two diffusion perf guards on H100 CI behave in ways we cannot explain from logs alone, and we have no Hopper machine outside CI.

1. qwen_image_t2i_cache_dit_enabled, steps 10/49. Since [Diffusion] Optimize Qwen-Image-Edit attention on Hopper #38584 (fused QK-norm/RoPE epilogue on Hopper), the first cached step after each full step measures ~64 ms instead of ~56 ms, while full steps dropped from ~188 to ~175 ms and total denoise time did not get worse. Step times are host wall times with no device sync (unless SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1), and the cache-dit residual check is the first blocking point after a full step. Hypothesis: the "slow" cached step is the previous full step's queued GPU work, and the fused epilogue only made the host enqueue faster.
2. ltx_2_5_diffusion_decoder_2gpus. With the DiT on component-offload, each server launch randomly runs denoise 1.4-1.9x slower. Clocks, load-time memory and logs are identical between fast and slow launches.

What this PR does

It changes only .github/workflows/pr-test-multimodal-gen.yml, so only NVIDIA diffusion CI runs. All diffusion jobs are disabled except two debug jobs. Each job applies a small env-gated instrumentation patch (below) to its own checkout, then launches one case repeatedly with plain pytest (no harness retries):

• 1-GPU H100 x1, qwen (14 launches): fused epilogue on/off (SGLANG_ENABLE_FUSED_QKNORM_ROPE) x synchronized step timing on/off, 3 interleaved launches each, plus one torch.profiler launch per fused setting covering steps 8-12.
• 2-GPU H100 x2, ltx_2_5 (8 launches per runner): A = CI config; D = same, but the DiT is loaded on the current stream instead of the component-offload prefetch stream. Every launch profiles steps 3-4 on both ranks.
• Every launch records per-rank step wall times and allocator counters (num_alloc_retries, num_device_alloc/free). A background nvidia-smi dmon records clocks, power, PCIe and utilization. Everything is uploaded as qwen-debug-* / ltx25-debug-* artifacts.

Expected footprint: one 1-gpu-h100 runner for ~45 min and two 2-gpu-h100 runners for ~75 min.

Instrumentation patch (applied in the job checkout only)

diff --git a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py b/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py
index 7481a5b41f..65686d5ce5 100644
--- a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py
+++ b/python/sglang/multimodal_gen/runtime/layers/attention/backends/sdpa.py
@@ -152,6 +152,9 @@ class SDPAImpl(AttentionImpl):
         ):
             # Keep the existing Flash SDPA arithmetic while consuming all
             # packed windows in one call, including ragged and empty windows.
+            if not getattr(SDPAImpl, "_dbg_varlen_logged", False):
+                SDPAImpl._dbg_varlen_logged = True
+                print("[DBG] sdpa varlen fast path used", flush=True)
             # Call the Flash op directly: from torch 2.14, varlen_attn silently
             # routes sm90/sm100 to cuDNN, which changes outputs versus the Flash path.
             output, *_ = torch.ops.aten._flash_attention_forward(
diff --git a/python/sglang/multimodal_gen/runtime/managers/memory_managers/component_residency_strategies.py b/python/sglang/multimodal_gen/runtime/managers/memory_managers/component_residency_strategies.py
index 63a5d52939..0426e69460 100644
--- a/python/sglang/multimodal_gen/runtime/managers/memory_managers/component_residency_strategies.py
+++ b/python/sglang/multimodal_gen/runtime/managers/memory_managers/component_residency_strategies.py
@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import os
 from typing import TYPE_CHECKING
 
 import torch
@@ -168,7 +169,7 @@ class ComponentOffloadStrategy(ComponentResidencyStrategy):
         use: ComponentUse,
         state: ResidencyState,
     ) -> bool:
-        if not current_platform.is_cuda():
+        if not current_platform.is_cuda() or os.environ.get("SGLANG_DEBUG_NO_PREFETCH_STREAM") == "1":
             self.prepare_for_use(module, use, state)
             return True
         if _module_ready_on_local_device(module, dtype=use.target_dtype):
diff --git a/python/sglang/multimodal_gen/runtime/models/dits/qwen_image.py b/python/sglang/multimodal_gen/runtime/models/dits/qwen_image.py
index 7b7ca0cc7c..bd6b2aa13b 100644
--- a/python/sglang/multimodal_gen/runtime/models/dits/qwen_image.py
+++ b/python/sglang/multimodal_gen/runtime/models/dits/qwen_image.py
@@ -1269,6 +1269,9 @@ class QwenImageCrossAttention(nn.Module):
                 self.norm_q.variance_epsilon,
                 self.norm_added_q.variance_epsilon,
             )
+            if not getattr(QwenImageCrossAttention, "_dbg_fused_logged", False):
+                QwenImageCrossAttention._dbg_fused_logged = True
+                print(f"[DBG] fused qkv epilogue called, returned={joint_qkv is not None}", flush=True)
 
         if joint_qkv is None:
             # Fused ModelOpt FP8 projections expose zero-copy Q/K/V views into
diff --git a/python/sglang/multimodal_gen/runtime/utils/perf_logger.py b/python/sglang/multimodal_gen/runtime/utils/perf_logger.py
index 2c103897a4..503f44fcb6 100644
--- a/python/sglang/multimodal_gen/runtime/utils/perf_logger.py
+++ b/python/sglang/multimodal_gen/runtime/utils/perf_logger.py
@@ -281,6 +281,105 @@ class RequestPerfRecord:
         self.memory_snapshots = memory_snapshots or {}
 
 
+_DBG_STEP_RE = None
+_DBG_PROF = {"active": None, "n": 0}
+
+
+def _dbg_sync_enabled() -> bool:
+    return (
+        os.environ.get("SGLANG_DIFFUSION_SYNC_STAGE_PROFILING", "0") == "1"
+        and torch.cuda.is_available()
+    )
+
+
+def _dbg_rank() -> int:
+    try:
+        import torch.distributed as dist
+
+        return dist.get_rank() if dist.is_initialized() else 0
+    except Exception:
+        return 0
+
+
+def _dbg_step_index(name: str):
+    global _DBG_STEP_RE
+    import re
+
+    if _DBG_STEP_RE is None:
+        _DBG_STEP_RE = re.compile(r"^denoising_step_(\d+)$")
+    m = _DBG_STEP_RE.match(name)
+    return int(m.group(1)) if m else None
+
+
+def _dbg_profile_range():
+    spec = os.environ.get("SGLANG_DEBUG_PROFILE_STEPS")
+    if not spec or not os.environ.get("SGLANG_DEBUG_PROFILE_DIR"):
+        return None
+    a, _, b = spec.partition("-")
+    return int(a), int(b or a)
+
+
+def _dbg_on_enter(name: str) -> None:
+    step = _dbg_step_index(name)
+    rng = _dbg_profile_range()
+    if step is None or rng is None or _DBG_PROF["active"] is not None:
+        return
+    if step == rng[0]:
+        prof = torch.profiler.profile(
+            activities=[
+                torch.profiler.ProfilerActivity.CPU,
+                torch.profiler.ProfilerActivity.CUDA,
+            ],
+            record_shapes=False,
+            with_stack=False,
+        )
+        prof.__enter__()
+        _DBG_PROF["active"] = prof
+
+
+def _dbg_on_exit(name: str, wall_s: float) -> None:
+    step = _dbg_step_index(name)
+    if step is None:
+        return
+    trace_dir = os.environ.get("SGLANG_DEBUG_STEP_TRACE_DIR")
+    if trace_dir:
+        import json
+
+        rec = {"rank": _dbg_rank(), "step": step, "wall_ms": wall_s * 1000.0}
+        if torch.cuda.is_available():
+            st = torch.cuda.memory_stats()
+            for k in (
+                "num_alloc_retries",
+                "num_device_alloc",
+                "num_device_free",
+                "num_sync_all_streams",
+                "num_ooms",
+            ):
+                rec[k] = st.get(k)
+            rec["allocated_gb"] = st.get("allocated_bytes.all.current", 0) / 2**30
+            rec["reserved_gb"] = st.get("reserved_bytes.all.current", 0) / 2**30
+        os.makedirs(trace_dir, exist_ok=True)
+        with open(
+            os.path.join(trace_dir, f"steps_rank{rec['rank']}.jsonl"), "a"
+        ) as f:
+            f.write(json.dumps(rec) + "\n")
+    rng = _dbg_profile_range()
+    prof = _DBG_PROF["active"]
+    if prof is not None and rng is not None and step == rng[1]:
+        torch.cuda.synchronize()
+        prof.__exit__(None, None, None)
+        _DBG_PROF["active"] = None
+        _DBG_PROF["n"] += 1
+        out = os.environ["SGLANG_DEBUG_PROFILE_DIR"]
+        os.makedirs(out, exist_ok=True)
+        tag = f"rank{_dbg_rank()}_occ{_DBG_PROF['n']}"
+        with open(os.path.join(out, f"kernels_{tag}.txt"), "w") as f:
+            f.write(
+                prof.key_averages().table(sort_by="cuda_time_total", row_limit=60)
+            )
+        prof.export_chrome_trace(os.path.join(out, f"trace_{tag}.json"))
+
+
 class StageProfiler:
     """
     A unified context manager, records performance metrics (usually of a single Stage or a step) into a provided RequestMetrics object (usually from a Req).
@@ -327,6 +426,10 @@ class StageProfiler:
             torch.get_device_module().synchronize()
 
     def __enter__(self):
+        _dbg_on_enter(self.stage_name)
+        if _dbg_sync_enabled():
+            torch.cuda.synchronize()
+        self._dbg_t0 = time.perf_counter()
         if self.log_stage_start_end:
             msg = f"[{self.stage_name}] started..."
             if self.logger.isEnabledFor(logging.DEBUG):
@@ -350,6 +453,10 @@ class StageProfiler:
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
+        if exc_type is None:
+            if _dbg_sync_enabled():
+                torch.cuda.synchronize()
+            _dbg_on_exit(self.stage_name, time.perf_counter() - self._dbg_t0)
         if not (
             (self.log_timing and self.metrics)
             or self.log_stage_start_end

Round 3: controlled A/B experiments

Rounds 1–2 left several conclusions resting on correlations in CI history. This round tests each one by changing a single factor on a single runner, interleaved and repeated. It only measures and changes no thresholds. A small env-gated patch to perf_logger.py records per-rank step wall time, thread CPU time, the CPU core and involuntary context switches, and optionally profiles named stages or steps. A host sampler and nvidia-smi dmon run throughout.

Exp	Runner	Question	Design
A	1-gpu-h100	Does #40470's conditioning cache slow longlive2_i2v, and is that real time or only time moved between stages?	(1) Current main, cache on vs --disable-conditioning-cache, each with unsynced (CI) and synced stage timing, ×3 interleaved, plus one stage profile of VAE encoding and decoding per arm. (2) Parent of #40470 vs #40470, A-B-A, reinstalling at each commit.
B	1-gpu-h100	Does #38641 (torch 2.14) slow the Wan 5B decoding stage?	9631f6f (torch 2.13) vs 0bdd0cc (#38641) A-B-A; wan2_2_ti2v_5b and fastwan2_2_ti2v_5b, synced and unsynced, plus a decoding-stage profile per block.
C1	2-gpu-h100	Does #38641 slow LTX two-stage denoise?	Same A-B-A; ltx_2_two_stage_t2v and ltx_2_3_two_stage_ti2v_2gpus ×2 per block, plus a steps 3–4 profile per block.
C2	2-gpu-h100	Is LTX two-stage about 14% above its baseline because of a regression or because of the baseline itself?	1456eb6 (#28123, which set these baselines on 06-14) vs current main, A-B-A, ×2 per case per block.
D	2-gpu-h100	At what input size does ltx_2_5 denoise stop being host-bound, and does it still fit with component-offload?	Current main at 768x448x49, 1280x704x49, 1280x704x97 and 1280x704x121, ×3 interleaved, with steps 3–4 profiled once per size. The output-consistency check is skipped because there is no reference output at the new sizes.

Expected footprint: two 1-gpu-h100 runners for about 1 h, and three 2-gpu-h100 runners for about 1–1.5 h. Artifacts are uploaded as dbg3-<exp>.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #37482923115
Latest PR Test (Extra): ❌ Run #37482922394
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[niehen6174]

Debug run finished (run 37266351867); artifacts downloaded. Closing.

[niehen6174]

Round 2 cancelled while still queued behind a large batch of re-runs; closing for now. Round 1 results are summarized in the description.