Revert "[DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement" (#45863)

[vllm-agent]

Revert of #45863

This reverts commit 0a7bacd (merge commit of PR #45863).

Reason: CI nightly build #72766 detected a critical accuracy regression in the MoE Refactor Integration Test (B200 - TEMPORARY) job. The test test_gsm8k_correctness[DeepSeek-V4-Flash-deepgemm-mega-moe] reported GSM8K accuracy of 6.75%, far below the expected threshold of 87%.

PR #45863 changed DSv4 flashinfer sparse index cache files (vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py, vllm/v1/attention/backends/mla/sparse_swa.py) which are directly related to the DeepSeek-V4 inference path.

Note: PR #45309 (which also changed DSv4 attention code) has already been reverted separately in #45972.

Linked failures (1):

• MoE Refactor Integration Test (B200 - TEMPORARY) — GSM8K accuracy 0.0675 < 0.87

Auto-generated by CI failure analyzer.

[yewentao256]

Retested this in main and it is not the reason

=========================== 1 passed, 16 warnings in 384.38s (0:06:24) ============================
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
(yewentao256) [yewentao256@ip-172-31-44-248 vllm-source]$  pytest -s -v tests/evals/gsm8k/test_gsm8k_correctness.py --config-list-file=tests/evals/gsm8k/configs/moe-refactor/config-b200.txt