Repository navigation
[RDNA2] INT4 PLE table on the upstream PLE path (VLLM_PLE_QUANT_DIR) - #49
Merged
Merged
Conversation
…he upstream PLE path Port of the fork's int4 PLE sidecar table onto upstream's PLE embedding. VLLM_PLE_QUANT_DIR pointing at a ples_int4 directory (META.json layout group16_int4_fp16scale_lownibblefirst, shard_<n>.safetensors with weight_i4 and weight_scale) selects Qwen4ExpPLEInt4EmbeddingMethod: each rank copies its vocabulary rows of the packed table and group scales into the existing pinned-host (or device) storage, the checkpoint's BF16 table is skipped, and lookups dequantize to the activation dtype (Triton UVA kernel for the pinned table, torch gather for the device table). Flash-Next: 30 GB table instead of 95 GB BF16. Unset, nothing changes; other layouts are rejected. Tests: tests/models/qwen4_exp/test_ple.py 73/73 (load across shard boundaries + dequant, layout rejection, pinned UVA kernel incl. rows owned by another rank), test_ple_amd.py 8/8 on gfx1030.
Optional script run against the live server after the bench cells (URL, PORT, SERVED, MODEL, OUT exported), e.g. tools/agentic_bench coding-agent traffic; its PASS/FAIL line lands in the summary.
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serves the Flash-Next PLE n-gram table as INT4 on the upstream PLE path.
Upstream's PLE offload keeps the table in BF16 (95 GB of host RAM). With
VLLM_PLE_QUANT_DIR=/path/to/ples_int4(primitive-ai/Qwen3.8-Flash-Next-PLE-quant), each rank loads its rows of the INT4 group-16 table (30 GB total) into upstream's pinned storage and dequantizes on lookup. Unset, nothing changes.Changes
Qwen4ExpPLEInt4EmbeddingMethod+ Triton UVA lookup kernel (common/ngram_embedding.py). Other table layouts are rejected.serve_validate.sh:LOAD_HOOKto run real-traffic workloads against the live server.Tests:
test_ple.py73/73 (3 new),test_ple_amd.py8/8 on gfx1030.Validation (4× V620, Flash-Next W4A16, TP=4, F&P): gates, 4 cells, stall probe, agentic traffic (1k c=8, 16k c=6, 32k c=4).
Cells: prefill tok/s/req · decode tok/s/req · TTFT s · ITL median ms · aggregate output tok/s.
Speed at parity with BF16; host RAM −65 to −70 GB. Stall probe unchanged (16k injection max gap 503 ms MTP0 / 548 ms MTP2).