[NPU] Support DSV4 host memory cache management

[cx22757]

Motivation

DeepSeek-V4 on NPU maintains physically independent FULL, SWA, C4, and C128 cache regions. The existing hierarchical-cache path primarily assumed GPU-style KV-derived sidecars and global page geometry, so it could not correctly manage all DSV4 NPU cache components during host backup, device eviction, and
load-back.

This PR adds DSV4 NPU host-memory cache management, including Ascend host/device transfers, independent C128 page ownership and capacity management, and correct handling of C4 indexer K/scale data.

Modifications

DSV4 NPU host pools and data transfer

• Add kernel_ascend D2H/H2D transfer support for DSV4 paged KV pools and compressor-state pools.
• Support layer_first, page_first, and page_first_direct host-memory layouts.
• Resolve host allocation functions using the NPU device type, avoiding CUDA host-registration fallback on NPU.
• Use the physical kernel_page_size for NPU C4 and C128 host pools.
• Convert KV-derived transfer indices according to each compressed pool's page ratio.
• Wait for pending per-layer cache transfers before exposing DSV4 KV buffers to computation.

C128 host-cache lifecycle

• Treat NPU C128 as an independent radix-cache component instead of a KV-derived sidecar.
• Add C128 host backup and load-back transfers for complete physical page groups.
• Track C128 host residency, host locks, LRU membership, device refcounts, and host-slot release.
• Restore request-local C128 page tables after host load-back.
• Size the C128 host pool from the independent C128 device pool.
• Check exact per-request C128 demand during extend, decode, and load-back.
• Reclaim C128 capacity through FULL leaf eviction so C128 pages remain consistent with their FULL-token ownership.

C4 indexer and cache-stack integration

• Store NPU C4 indexer int8 K data and fp16 scale data in separate typed host pools.
• Register the scale pool with the hierarchical-cache stack and Mooncake storage.
• Preserve the existing packed indexer representation on GPU.
• Extend the allocator capacity interface with request and speculative-algorithm context while preserving the default behavior for other allocators.
• Pass request-level context from decode scheduling so DSV4 can calculate C128 page demand before allocation.

Accuracy Tests

Accuracy was evaluated on the complete GPQA-Diamond dataset with hierarchical cache enabled. The validation used one Ascend node with 16 NPUs in TP16/DP16 mode. Speculative decoding and DSpark were not enabled.

• model: DeepSeek-V4-Flash-w8a8-mtp
• Hardware: 16 A3 NPU
• Reproducer dataset: GPQA Diamond, 198 examples

Configuration	GPQA scores from 3 runs	Mean
NPU with L1/L2 HiCache	0.868, 0.873, 0.894	0.878

Speed Tests and Profiling

Serving latency was evaluated on the same Ascend node with hierarchical cache enabled
(16 NPUs, TP16/DP16). Speculative decoding and DSpark were not enabled. Each row reports
the mean TTFT (time to first token) over 16 concurrent deterministic requests with
round-robin routing across the 16 DP ranks (output length 1).

• Workload: deterministic synthetic prefixes, 16 concurrent, output length 1, round-robin DP16

Request length	Hit rate	TTFT (cold prefill, ms)	TTFT (L1 hit, ms)	TTFT (L2 hit, ms)	L1 speedup (×)	L2 speedup (×)
64K	100%	45504.4	457.1	2107.8	99.6	21.6
128K	100%	115115.0	595.1	3970.4	193.5	29.0
256K	100%	306020.9	968.2	6266.5	316.1	48.8

The following non-speculative server configuration was used for DSV4 NPU L1/L2 HiCache validation. Machine-specific paths and network settings are exposed as environment variables.

Click to expand: single-node 16-NPU launch command

#!/usr/bin/env bash
set -euo pipefail

MODEL_PATH=${MODEL_PATH:-/path/to/DeepSeek-V4-Flash-w8a8-mtp}
SGLANG_ROOT=${SGLANG_ROOT:-/path/to/sglang}
ASCEND_TOOLKIT_ROOT=${ASCEND_TOOLKIT_ROOT:-/usr/local/Ascend/ascend-toolkit/latest}
RUN_DIR=${RUN_DIR:-${PWD}/dsv4_hicache_run}
HOST=${HOST:-0.0.0.0}
PORT=${PORT:-30000}

mkdir -p "${RUN_DIR}"

source "${ASCEND_TOOLKIT_ROOT}/opp/vendors/customize/bin/set_env.bash"
source "${ASCEND_TOOLKIT_ROOT}/opp/vendors/custom_transformer/bin/set_env.bash"

export ASCEND_RT_VISIBLE_DEVICES="${ASCEND_RT_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}"
export HCCL_SOCKET_IFNAME="${HCCL_SOCKET_IFNAME:-lo}"
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1500
export HCCL_OP_EXPANSION_MODE=AIV
export INF_NAN_MODE_FORCE_DISABLE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32

export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_DSV4_FP4_EXPERTS=False
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False
export SGLANG_OPT_FP8_WO_A_GEMM=0
export SGLANG_OPT_FUSE_WQA_WKV=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=False
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False
export SGLANG_OPT_USE_TILELANG_MHC_POST=False
export SGLANG_OPT_USE_TILELANG_MHC_PRE=False
export SGLANG_SET_CPU_AFFINITY=1

cd "${SGLANG_ROOT}"

exec python3 -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --page-size 128 \
    --tp-size 16 \
    --dp-size 16 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --trust-remote-code \
    --device npu \
    --attention-backend ascend \
    --quantization modelslim \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --kv-cache-dtype auto \
    --watchdog-timeout 9000 \
    --host "${HOST}" \
    --port "${PORT}" \
    --mem-fraction-static 0.7 \
    --swa-full-tokens-ratio 0.5 \
    --prefill-max-requests 1 \
    --chunked-prefill-size 32768 \
    --max-running-requests 16 \
    --context-length 1048576 \
    --max-total-tokens 532480 \
    --random-seed 20260824 \
    --enable-hierarchical-cache \
    --hicache-io-backend kernel_ascend \
    --hicache-ratio 2.0 \
    --hicache-write-policy write_through \
    >>"${RUN_DIR}/server.log" 2>&1

---

CI States

Latest PR Test (Base): ✅ Run #34562923858
Latest PR Test (Extra): ❌ Run #34562923714
Latest PR Test (AMD ROCm 10): ❌ Run #34562923886

[cx22757]

/rerun-failed-ci

[cx22757]

补充L2从L1 load back命中相比L1直接命中，性能劣化情况

[sglang-npu-bot]

/tag-run-ci-label

[sglang-npu-bot]

/tag-and-rerun-ci

[cx22757]

/rerun-failed-ci

[cx22757]

/rerun-failed-ci

[cx22757]

/rerun-failed-ci

[cx22757]

/rerun-failed-ci

[cx22757]

/rerun-failed-ci

[cx22757]

/rerun-failed-ci

[cx22757]

base-c-test-perf-16-npu-a3: this failure case was introduced by 32495.