fix(comm): preserve PCIe IPC tuning under unrelated outer autotune

[ormandj]

📌 Description

This is a focused follow-up to flashinfer-ai#4393.

A caller can wrap model warmup in a FlashInfer autotune context that replaces the singleton tuner's active file cache without installing the distributed process group required to profile the PCIe-IPC collective.

Previously, PcieIpcAllReduceWorkspace treated any active autotune context as permission to call choose_one(). In an unrelated outer context, the workspace could lose its explicitly loaded tuning cache or retain the seed launch configuration.

This change:

• exposes the runner's collective-safe can_profile() predicate;
• profiles candidates only when autotune is active and the required distributed tuning group is installed;
• reloads the workspace's explicit tuning cache when an unrelated outer autotune context replaced the singleton cache; and
• performs the normal cache lookup using the workspace's own bucket policy.

Serving and autotune validation

The change was exercised in the actual SGLang serving stack, not only through a mocked unit test.

The tested configuration used:

• two RTX PRO 6000 Blackwell Max-Q GPUs, SM120, TP2 over PCIe Gen4 x16;
• DeepSeek-V4-Flash-0731 with DSPARK block size 5;
• SGLang's enclosing FlashInfer model-warmup autotune context;
• a persisted PCIe-IPC tuning cache;
• SGLANG_PCIE_IPC_MAX_NUMEL=786432;
• CUDA graph capture and replay; and
• clients running inside the serving pod against localhost.

Both TP ranks initialized PCIe-IPC successfully, the tuned serving configuration remained active after the enclosing autotune context, and the server completed the complete release qualification.

Matched backend profiling

The following measurements establish that the parent PCIe-IPC backend was exercised by the serving stack. They are not presented as isolated performance attribution for this cache-preservation follow-up.

The NCCL and PCIe-IPC captures used matched source, image, model arguments, graph sizes, token IDs, cache policy, and profiler settings. Each result retained 127 complete paired target-plus-draft graph steps.

Workload	NCCL step	PCIe-IPC step	Change
C1	15.681 ms	14.965 ms	-4.57%
C8	35.399 ms	33.421 ms	-5.59%

At C8, all 11,938 captured all-reduce operations used PCIe-IPC. The per-rank timeline reductions were:

Rank	Time removed per step	Share of NCCL step
0	1.734 ms	4.90%
1	1.781 ms	5.03%

Prefill-sized reductions remained on NCCL.

Integration qualification

The source-equivalent release stack containing the outer-autotune fix produced these five-repetition medians:

Concurrency	Forward passes/s
C1	63.864
C2	47.737
C4	34.322
C8	22.968
C16	17.304
C32	12.722

Cold-prefill aggregate results, with five requests per length:

Prompt length	Prompt tok/s
8K	7,776.8
32K	8,743.1
64K	8,464.5
128K	7,966.6

Quality and long-output results:

• GSM8K: 1,261/1,319 correct, zero request errors.
• Long output: 8/8 HTTP successes with finish_reason=stop.
• Structural validation: 8/8 passed all configured checks.

These are integration results for the complete serving stack. They are not attributed solely to this follow-up.

🔍 Related Issues

• Parent implementation: [feat]custom all reduce kernel flashinfer-ai/flashinfer#4393
• SGLang consumer: [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts sgl-project/sglang#34528

🚀 Pull Request Checklist

✅ Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit or used an equivalent environment.
• I have installed the hooks with pre-commit install.
• I have run the hooks manually with pre-commit run --all-files.

🧪 Tests

• Tests have been added or updated as needed.
• The focused tests pass.
• The change has been exercised under the real SGLang outer-autotune serving path.

Focused tests:

uv run --no-project pytest -q tests/comm/test_pcie_ipc_policy.py tests/comm/test_pcie_ipc_tuning.py

The regression test verifies that an unrelated active autotune context reloads the workspace cache, performs one cache lookup, selects the cached tactic, and does not attempt unsafe collective profiling.

Reviewer Notes

This follow-up does not change PCIe-IPC kernels, topology policy, candidate generation, or the parent PR's tuning algorithm. It only distinguishes a collective-safe tuning context from an unrelated process-global autotune context.

Prepared with AI assistance.

[qsang-nv]

Close this PR as its content is cherry picked in flashinfer-ai@830f79e. Thanks for the contribution!