[AMD] Fix FP4 indexer OOR

[RolaoDenthu]

Motivation

On MI355X, DeepSeek-V4-Pro with the FP4 indexer aborts under long-running agentic workloads:

HSA_STATUS_ERROR_OUT_OF_RESOURCES ... Available Free mem : 0 MB
  at fused_moe_triton_kernels.py:1353 fused_append_shared_experts

The indexer allocates a fresh torch.empty(num_tokens, max_seq_len) for logits on every layer of every forward. If no cached block is large enough to reuse, it has to allocate a new one. Torch does release the tensor when it's done, but only back to its own caching allocator — the driver still sees that memory as taken. And max_seq_len follows the context length, which in an agentic session only grows, so each request is slightly larger than the last, the older blocks never get a chance to be reused, and they just pile up in the pool. Driver-visible memory drains block by block until the allocation fails with out of resources.

Modifications

1. Pooled fixed-size logits buffer (fp4_indexer_hip.py)

Prefill logits are served as a view into one pre-allocated block instead of a fresh torch.empty per call. The request size becomes constant, so the block is always reused and nothing strands. Decode keeps the plain allocation: its rectangle is bounded by the capture shapes and lives in the graph memory pool, and creating the pooled block mid-capture would hand graph-pool memory to later replays. Budget is set by SGLANG_DSV4_FP4_LOGITS_BUDGET_MB (default 2048).

2. Row chunking (indexer.py)

A fixed block alone isn't enough — a batch that doesn't fit would fall back to a dynamic allocation and start stranding again. logits_rows_per_chunk() reports how many rows fit, and the FP4 prefill path loops over row chunks, running each chunk's top-k before reusing the block. This decouples peak scratch from context length. Rows are scored and reduced independently, so the result is elementwise-identical to a single pass; the single-chunk case (the common one) passes the whole batch through without slicing. The KV payload/scale getters are hoisted out of the loop since they await the layer's KV transfer, and topk_v2 metadata is re-planned per chunk because the cached plan routes rows by their index in the full range.

Accuracy Tests

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #33735112082
Latest PR Test (Extra): ❌ Run #33735111880
Latest PR Test (AMD ROCm 7.2): ⏳ Run #33735112202

[1am9trash]

This PR

• Pre-allocate a large buffer up front and reuse it. Only reallocate when the decode req exceeds the current capacity. This avoids repeated alloc/free cycles that cause fragmentation.
• When the required size is larger than buffer supported (e.g. prefill with long ctx), process the rows in chunks.

LGTM.

[HaiShaw]

Changes hip gated. LGTM.