Port measured V620 fast serving stack onto current rdna_extras

[GeorgeMA-Strong]

Purpose

Port the fast V620 TP4 serving stack onto current upstream rdna_extras (f3dd65fa70636566c11b249209d281c0b63c819e) through Git commits. The server built and pulled this branch from Git; no source files were transferred over SSH.

The base already contains PR #15's live-context QSA bound and earlier FP16 tuning.

Measured 16K prefill

llm-context-bench ran regular and coding 16K fixtures on the same four V620 cards, TP4/EP4, FP16, MTP2, with warm passes and fresh request tags.

Stage	Regular tok/s	Coding tok/s
Exact upstream base	1,411–1,424	1,464–1,478
Resident W4A16 + FP16 shared expert + dequant/slot fixes	1,944–1,948	1,951–1,955
Qualified eight-row MoE tile	2,000–2,003	2,000–2,009
Fused PLE grouped normalization	2,014–2,016	2,013–2,022
Prefix checkpoint replay fixes	1,949–1,950	1,967–1,969
Served MoE table and launcher settings	1,951–1,952	1,970–1,972
Served skinny decode enabled	1,947–1,948	1,966–1,967

Long-context defect and fix

The 3.75 GiB service initially entered a deterministic preemption loop during the 143,855-token coding prompt. Scheduler instrumentation proved the allocation itself was not the cause. Four Mamba managers allocated a new state block per 4,096-token chunk but did not retire their previous states across null gaps. At 135,168 computed tokens only three of 278 blocks remained; the next reservation failed and restarted the only request from zero.

Commit b619cf991 backports mainline vLLM commit fadfe1c7d, [Bugfix][Core] Retire Mamba states across null gaps (#55450), while preserving the served branch's checkpoint/replay implementation. The focused server tests passed: 7 passed. The instrumented 128K coding request then crossed the former failure boundary with 123 blocks free before allocation and 119 after it, and completed at 1,832.6 prefill / 70.2 decode tok/s. Temporary scheduler instrumentation was removed in 5030766bc and b88acd94d.

Final Git service verification

The clean service retains the intended 3.75 GiB KV allocation (4026531840) and 602,112-pixel vision cap. Clean startups took 352 and 367.5 seconds, compared with 373.879 seconds for the saved combined service. The pinned llm-context-bench harness completed with retries disabled:

Case	Prompt tokens	Prefill tok/s	Decode tok/s	TTFT s	Valid
Regular 32K	33,455	1,977.2	67.5	16.92	Yes
Coding 32K	n/a	n/a	n/a	n/a	No: dominant repeated token
Regular 64K	66,913	1,932.5	56.6	34.63	Yes
Coding 64K	71,971	1,928.0	69.8	37.33	Yes
Regular 128K	133,816	1,823.5	54.9	73.38	Yes
Coding 128K	143,855	1,813.9	78.2	79.31	Yes

Exact PR-head rerun

After restarting directly from final PR SHA 532ed3f930f475e731c82726e079f3374ab5b33f, llm-context-bench completed both 16K fixtures with retries disabled:

Case	Prompt tokens	Prefill tok/s	Decode tok/s	TTFT s	Valid
Regular 16K	16,750	1,957.8	69.7	8.56	Yes
Coding 16K	18,063	1,982.8	68.5	9.11	Yes

Raw result: /home/george/v620-experiments/git-final-pr-head-16k-20260923.json. The live process has the PR checkout as its working directory and PYTHONPATH, imports vllm/__init__.py from that checkout, and maps _C, _rocm_C, _C_stable_libtorch, and _moe_C_stable_libtorch from that checkout's build/ directory. The tracked server tree is clean.

Vision and long-chat checkpoint validation

The final PR-head service passed real API probes for vision and appended-turn reuse. Each reuse probe used a unique cache salt and failed unless at least 70% of the prior prompt was cached and follow-up TTFT fell below half of the initial turn.

Conversation	Initial prompt	Initial TTFT	Follow-up prompt	Cached on follow-up	Reused	Follow-up TTFT
Text 32K	33,628	16.88 s	33,648	31,744	94.3%	1.26 s
Text 128K	126,028	68.03 s	126,048	124,928	99.1%	4.11 s
Vision + text 35K	35,151	17.69 s	35,171	33,792	96.1%	3.14 s

The vision probe used a generated 1,260×1,260 RGB image (1,587,600 pixels), close to the configured 602,112-pixel cap. The model correctly identified its blue and yellow regions and central white outline. The long multimodal probe kept that image at the beginning of the conversation, appended the assistant answer and another user turn, and returned the required answer while reusing 33,792 prior tokens.

Raw server artifacts:

• /home/george/v620-experiments/prefix-reuse-final-32k-20260923/results.json
• /home/george/v620-experiments/prefix-reuse-final-128k-20260923/results.json
• /home/george/v620-experiments/vision-final-1260-followup-20260923.json
• /home/george/v620-experiments/vision-longchat-final-32k-20260923.json

CUDA graph scope

The measured launcher has used mode=0, cudagraph_mode=FULL_DECODE_ONLY, and capture sizes [3,6,12] since baseline commit 9daf24d68. With MTP2, each request schedules three uniform decode tokens, so those sizes cover 1, 2, and 4 simultaneous requests. The final live startup log confirms that all four workers captured FULL graphs for those descriptors. Prefill and mixed batches execute without CUDA graphs.

PIECEWISE is not active in this configuration: this branch requires compilation mode VLLM_COMPILE (mode=3) and piecewise graph partitions for it. Historical September 15 Flash-Next testing found 11/18 post-load correctness failures with FULL_AND_PIECEWISE plus prefix caching, while decode-only capture remained the qualified served path. The current PR includes later graph and recurrent-state fixes, so that historical result is evidence for retaining the measured setting rather than a claim that current FULL_AND_PIECEWISE can never work. Switching modes requires a separate current-branch correctness and performance A/B.

Exact full run command

This is the resolved command for the measured service. It includes every serving, ROCm, cache, TunableOp, and V620 environment variable from the live process. The OS-managed systemd variables are supplied by the user service. The service unit also applies MemoryMax=236223201280, MemorySwapMax=0, KillMode=control-group, and Restart=on-failure.

set -euo pipefail

repo=/home/george/v620-experiments/baseline-git-20260922
runtime=/home/george/v620-experiments/upstream-20260915/v620-vllm-testing
model=/home/george/v620-vllm/models/intel-autoround
branch=codex/v620-baseline-port-20260922
expected=532ed3f930f475e731c82726e079f3374ab5b33f

cd "$repo"
git fetch origin "$branch"
git switch "$branch"
git pull --ff-only origin "$branch"
test "$(git rev-parse HEAD)" = "$expected"
test -z "$(git status --porcelain --untracked-files=no)"

export V620_ENABLE_RESIDENT=1
export V620_ENABLE_SKINNY=1
export V620_HOST=0.0.0.0
export V620_PORT=8080
export PYTHONPATH="$repo"
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export VLLM_PLE_CPU_OFFLOAD=1
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_MOE_PREFILL=0
export VLLM_GDN_HIP_PREFILL=0
export VLLM_RDNA_FUSED_SE=1
export VLLM_TUNED_CONFIG_FOLDER="$repo/tuned-moe"
export VLLM_RDNA_MOE_RESIDENT=1
export VLLM_RDNA_MOE_RESIDENT_SKINNY=1
export VLLM_RDNA_DENSE_INT8=0
export VLLM_RDNA_DENSE_INT8_ONLY=0
export VLLM_RDNA_DENSE_GEMV=0
export VLLM_RDNA_AR=1
export VLLM_RDNA_AR_MAX_KB=64
export VLLM_RDNA_AR_BLOCKS=0
export VLLM_RDNA_AR_PACE=0
export HSA_FORCE_FINE_GRAIN_PCIE=1
export HSA_ENABLE_SDMA=0
export OMP_NUM_THREADS=4
export TOKENIZERS_PARALLELISM=false
export PYTHONFAULTHANDLER=1
export VLLM_CAUSAL_CONV1D_RDNA2_FWD=0
export VLLM_CAUSAL_CONV1D_RDNA2_UPDATE=0
export VLLM_ENABLE_STARTUP_PLAN=0
export VLLM_ROCM_USE_AITER=0
export TORCH_BLAS_PREFER_HIPBLASLT=0
export VLLM_CACHE_ROOT="$repo/cache/vllm"
export TRITON_CACHE_DIR="$repo/cache/triton"
export TORCHINDUCTOR_CACHE_DIR="$repo/cache/inductor"
export TORCH_EXTENSIONS_DIR="$repo/cache/extensions"
export PYTORCH_TUNABLEOP_ENABLED=1
export PYTORCH_TUNABLEOP_TUNING=0
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export PYTORCH_TUNABLEOP_FILENAME="$runtime/tunableop/rocblas-c27e2252cc7a/tunableop_results.csv"
export PATH="$runtime/.venv/bin:/opt/rocm/core-10.0/bin:/opt/rocm/core-10.0/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"
export LD_LIBRARY_PATH="$runtime/.venv/lib/python3.12/site-packages/_rocm_sdk_core/lib:$runtime/.venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/host-math/lib:/opt/rocm/core-10.0/lib"

args=(
  -m vllm.entrypoints.openai.api_server
  --model "$model"
  --served-model-name active qwen3.8-flash-next
  --host 0.0.0.0
  --port 8080
  --tensor-parallel-size 4
  --pipeline-parallel-size 1
  --enable-expert-parallel
  --enable-ep-weight-filter
  --dtype float16
  --max-model-len 262144
  --block-size 1024
  --max-num-seqs 4
  --max-num-batched-tokens 4096
  --kv-cache-memory-bytes 4026531840
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[3,6,12]}'
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
  --enable-auto-tool-choice
  --tool-call-parser qwen3_xml
  --reasoning-parser qwen3
  --default-chat-template-kwargs '{"enable_thinking":false}'
  --limit-mm-per-prompt '{"image":255,"video":32}'
  --mm-processor-kwargs '{"max_pixels":602112}'
  --enable-prefix-caching
  --mamba-cache-mode align
  --kernel-config '{"moe_backend":"triton"}'
)

"$runtime/.venv/bin/python" "$repo/tools/rdna2/check_v620_tuning.py" \
  --rows-template "$PYTORCH_TUNABLEOP_FILENAME" "${args[@]}"
exec "$runtime/.venv/bin/python" "${args[@]}"

The checked-in unit and launcher execute the same configuration:

systemctl --user daemon-reload
systemctl --user enable --now v620-tp4-git.service

The resolved rocBLAS library SHA-256 is c27e2252cc7a4b6e174601a180a169822dcf15a29e634046daecb61498e3bd2a; the matching TunableOp directory suffix is c27e2252cc7a.

Verification

• Native gfx1030 ROCm build from the server Git checkout passed.
• Focused GPU tests passed for INT4 zero preservation, resident MoE, eight-row tile equivalence, recurrent slot bounds, PLE grouped norm, checkpoint replay, and skinny decode.
• The focused Mamba retirement tests passed on the server: 7 passed.
• Pre-commit checks, mypy, and Markdown lint passed on changed files.
• The Git service is active and healthy from this branch; the old fast unit is disabled and inactive.

Reproduction and scope

The benchmark log and per-stage results and Git-only build and service recipe record exact settings, model metadata hashes, runtime paths, startup timing, and result JSON paths. Model shards and the Python environment remain external machine assets.

AI assistance was used to port, test, and prepare this draft. Human review is requested before merge.