Update vllm version to support llama3.1 by Ying1123 · Pull Request #705 · sgl-project/sglang

Ying1123 · 2024-07-23T20:11:37Z

No description provided.

## Motivation  ## Modifications  ## Accuracy Tests  ## Benchmarking and Profiling  ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).

…3 us threshold Desk half of sgl-project#705, the sgl-project#485 phase matrix applied to DECODE as sgl-project#702 applied it to prefill. No GPU touched. Verdict: the desk-net is positive if and only if a blocking TP all-reduce costs more than 14.3 microseconds on this rig. Everything else is priced; the whole question collapses to that one measurable threshold, which is the only number this desk could not source. Two premise corrections, one each way. (a) "20 GB of GDN weights" is ~4x too high for the part that actually moves. The MoE block is 77.7 percent of all weights and does NOT move -- it stays sharded on every layer. Only the GDN attention module relocates: 110.5 MiB per layer, 5,304 MiB total. (b) "75 percent of depth becomes sync-free" is FALSE, and this cuts against the proposal. Every layer is MoE (qwen3_next.py:580, is_layer_sparse = True unconditionally) and the MoE block issues its own all-reduce whenever moe_tp_size > 1 (moe/fused_moe_triton/layer.py:2047, :2062). No layer becomes sync-free at any depth. The removal is 48 of 128 collectives per round, 37.5 percent, not 75 percent of depth. (c) Cutting FOR the proposal: the 48 removed all-reduces are the BLOCKING kind (RowParallelLinear calls tensor_model_parallel_all_reduce synchronously, linear.py:2340), while the surviving MoE all-reduce is DEFERRED and joined in the next layer's prepare_attn (communicator.py:888-891, sgl-project#597). So the removed collectives are worth more per unit than a count comparison suggests. Bandwidth cost, priced: +0.668 ms/round for GDN attention weights (sharded 1,768 MiB each binds on the 3080s at 2.439 ms; solo 5,304 MiB on the 5090 is 3.107 ms) plus +0.018 ms for state, so +0.686 ms/round total -- far below the ticket's ~+5 ms envelope. The reason is that the sharded case is bound by the SLOWEST rank, so trading three slow readers for one fast one costs much less than the aggregate-bandwidth framing implies. Break-even is 0.686 ms / 48 = 14.3 us per collective, a low bar for a PHB round trip on a 10 KB payload. Capacity ledger: rank0 +4,688 MiB (GDN weights +3,536, GDN states +1,152 at 12 mamba slots), each 3080 frees 2,344 MiB, world net EXACTLY ZERO. Here that conservation is correct rather than the sgl-project#702 error: decode really is the TP phase, the pool really is the sum, and the DCP vector really can relieve a tight rank. sgl-project#702's mistake was applying this rule to PP prefill, where the pool is layer-sharded and takes the min. Residual tension named: capacity wants rank0's token share lower, bandwidth wants 2.4:1:1. Recommendation is to hold 2.4:1:1 and absorb the 4,688 MiB out of rank0's KV share, since the world pool is conserved and the 3080s were just handed 2,344 MiB each. Ranks 1/2 become zero-shard for the GDN family per sgl-project#115, an intermediate rung rather than all-or-nothing, as sgl-project#324 per-(rank,family) ratios allow. Ledger gap stated rather than smoothed: the census on disk is the PP layout, so world-neutrality holds as an accounting identity but the per-rank FIT of +4,688 MiB on rank0 is unverified. One conditional A/B arm defined, gated on measuring the break-even first, with the wait column as the acceptance signal and a falsifier on rank0 compute rising by ~0.69 ms and no more. Must not be co-armed with a sgl-project#702 prefill-cut change -- one phase at a time or neither result is attributable. Docs only. No arm, no deploy.

…s, not numbers or env flags Binding directive (PLAN_PERF_PIPELINE_2026-08-16, "PLANNER-SOLVED, UNIVERSAL"): the family-split verdict must be an objective/constraint set so solve() picks family ratios on any hardware, and the ReplaySSM enable must be a planner-visible gated capability rather than a flag someone remembers to set. The pp_cut phase-rule work is the pattern: rules, not numbers. planner/family_split.py -- nothing in it names a GPU or a model. Collective costs arrive from the census instruments, bandwidths from the pair matrix, capacity from the ledger. The rules: * a family's BLOCKING collectives vanish iff its placement is concentrated on exactly one rank; partial concentration buys nothing, and DEFERRED collectives never vanish, so a family whose collectives are all deferred can never repay concentration; * sharded time depends on the shard POLICY -- equal width shard lets the slowest rank bind, bandwidth-proportional (uneven TP) has every rank finish together at bytes / sum(bandwidth); * concentration therefore pays only when it escapes a slow BINDING rank; * capacity is a hard per-rank constraint, and under the TP sum rule concentration redistributes rather than consumes; * an unmeasured collective cost is REFUSED, not guessed. Writing it as rules immediately falsified my own NOTE_705 headline. That note priced concentration against an equal 1/3 shard and derived a 14.3 us gate, but this fork ships --rank-tp-ratio, so the honest baseline is the proportional shard: 1.73 ms instead of 2.51 ms, and the gate roughly DOUBLES to 30.5 us. The solver derives both and a test pins the pair; the note carries a revision-2 correction block. The measured-cost fixture of 40 us still clears the corrected bar, so the recommendation is unchanged -- but it was closer than the note claimed. The generality test is the point of the exercise: on a uniform-bandwidth rig the same rules REJECT the placement they recommend here, because there is no slow rank to escape. A solver that still said "solo" would be fitting this rig. Capacity conservation, a slower host never being chosen, and the bandwidth-proportional fallback ratios are pinned alongside. planner/gated_capability.py -- ReplaySSM's enable becomes a decision with a printable reason. An UNMEASURED quality gate is a refusal, distinct from a measured failure ("we did not look" and "we looked and it diverges" support different decisions). An unmeasured net is a refusal. Byte-identical plus net above the floor enables; anything not byte-identical is lossy and refused unless lossy is explicitly permitted, and a large net does not override that ordering. The live ReplaySSM state -- net counted at 40.4 percent, identity gate never run -- is pinned as REFUSED, which is exactly where sgl-project#700 left it. Tests: 18, red first (11 + 7). Planner suite 2611 passed, 2 failed, both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2. ruff clean. No arm, no deploy.

…ptures the win instead The gate stopped being a guess: the on-rig collective cost is MEASURED at ar_10kb_us = 31.0-33.7 us for the 10 KB bs=1 payload (INTEGRATION_R3_VALIDATION __group__ rows; ANALYSE_321 uses 32.4). The 44.92 us in the tier tables is UCX to a REMOTE rig over 40G RoCE and does not apply. Total decode collective cost ~4.1 ms/round at 128 collectives/token; this proposal removes 48. The answer hinges entirely on which sharded baseline is honest: solo on 5090 3.192 ms EQUAL 1/3 shard (what runs today, ratio=None) 2.506 ms PROPORTIONAL (uneven TP, already shipped) 1.726 ms Removing 48 collectives saves 1.488-1.618 ms. Net vs today's equal shard is +0.802 to +0.932 ms; net vs the proportional shard is +0.022 to +0.152 ms. THE FINDING: uneven TP ALONE is worth +0.780 ms/round at zero capacity cost and no structural change -- it is already shipped and simply not enabled here (rank_tp_ratio=None). The family split BEYOND uneven TP is worth +0.090 ms, or about 0.3 percent of a ~30 ms bs=1 round, in exchange for concentrating a family on one rank, moving 4,688 MiB of residency, and taking on the sgl-project#115 zero-shard machinery. Not a trade worth a window. REFUSED; no A/B requested. Capacity ledger, priced against the measured TP-stack sizing pass (holdback 0.000 percent there, so these are real KV budgets): world total 1,432,230 tokens before and 1,432,230 after -- delta EXACTLY zero, every budget positive, rank0's share falling 45.0 to 34.5 percent. The split fits. Capacity was never the obstacle; the win was. The section-1 premise correction remains load-bearing: only 5,304 MiB of GDN ATTENTION relocates, not "20 GB". MoE is 77.7 percent of weights and stays sharded on every layer, which is also why no layer becomes sync-free and only 48 of 128 collectives are removable. If revisited, the order is fixed by the numbers: enable uneven TP first and re-measure. The split becomes interesting only if collective cost rises materially or the GDN attention family grows relative to MoE -- neither is true on this rig. Docs only. No arm, no window requested.

…olved attention family The remainder I sized in cc8e666 was two structural problems. Both are fixed, and the acceptance is the one that slice set: the function on the real checkpoint no longer returns {}. Qwen3.8-27B-INT8-yarn1.5 -> {'moe': 'int8', 'mlp': 'int8', 'vocab': 'bf16', 'attn_gdn': 'bf16'} CLASS-SELECTOR ATTRIBUTION, AS A COMPLEMENT. _is_class_selector recognises a bare module-class target (Linear) and distinguishes it from a path or a regex, which name specific modules and must not be widened. A class selector then contributes its key to every GEMM family the ignore list does NOT name, derived from _ALL_GEMM_FAMILIES rather than spelled out twice. The direction is the load-bearing choice: claiming a family the quantizer was told to skip would overwrite a bf16-resident fact with a scheme never applied. Two reverse pins hold it -- a regex target and a dotted path each stay narrow. THE ATTN_GDN SPAN, RESOLVED BY LAYERS. _weigh_attn_gdn_by_layers uses sgl-project#371's per-layer counts: linear_attn evidence weighs the GDN layers, the rest of the family weighs the full-attention ones. Reversing the split reverses the answer, which is what makes it a measurement rather than a vote -- _dominant's config-entry count never could. sgl-project#371 WAS REUSED, NOT REBUILT. Its census covers the block_configs axis; the GDN-hybrid axis was already derived from layer_types (uneven_perf.py:3712-3717, full_layers / gdn_layers). Nothing needed extending -- the per-layer truth existed and simply was not reaching this function. The caller now derives the same split and passes it. THE SHIPPED KEY IS A MAJORITY KEY, AND ITS PROVENANCE IS THE COMPLEMENT RULE. On this checkpoint attn_gdn resolves through the complement (its only evidence is the ignored linear_attn), NOT through the layer weighting. So attn_gdn=bf16 describes 48 GDN layers and does not describe the 16 full-attention ones. My first draft printed "resolved per layer" there, which overclaimed; the description now states the span and names the layers the key does not cover, so no consumer can read it as uniform. The per-layer VECTOR contract is filed as a named follow-up in NOTE_485_item1_family_report.md, with its consumer (sgl-project#324 per-(rank, family) scores and sgl-project#705-class family-placement pricing, which would otherwise charge full-attention layers at the GDN family's bf16 rate) and its feed (sgl-project#371's census, already measured). Not built here: it changes the interface every sgl-project#324 consumer reads and deserves its own red-first slice rather than being smuggled into a reporting fix. int8_a16 IS DECLARED, NOT LANE-REGISTERED. FORMATS_WITHOUT_LANES records weight-only INT8 as recognised but unmeasurable and rank_gemm_scores says so in its own branch. It is deliberately NOT added to _FORMAT_LANES: that table's own comment says registering a lane the serving path cannot take would make the plan lie, and this tree has no weight-only int8 arm. The fix is sgl-project#606's known-but-unmeasured versus unrecognised distinction, not a fabricated lane. 20 tests, red first. Mutation-proven: disabling class-selector recognition and the layer weighting together red 7, including all four real-checkpoint acceptance tests. 133 passed / 10 skipped across the planner consumer suites, up from 120 -- the delta is these tests. Mapping was missing from the typing import, caught by ruff and invisible at runtime under postponed annotations; uneven_perf.py is back to its 6 pre-existing findings. No boots.

…its second support The exhaustive measurement sweep landed after the first pass. Both verdicts survive; four things sharpen, one against me. "NO CUDA P2P" WAS FACTUALLY RIGHT. capability_matrix.json in the 2026-07-30 bar1 battery reports can_access_peer: false for all six directed pairs, and d2d_bench.json's near-identical direct vs staged timings follow from it. The error was the second clause: BAR1 does not go through CUDA P2P at all -- it maps the peer aperture via dmabuf and writes into it, which is why can_access_peer does not bound it. I read a true fact about one mechanism as a bound on another. THE MAGNITUDE ARGUMENT WAS ALSO WRONG, and this is the bigger error. benchmark/bench_host_transport.py:12 records a MEASURED point-to-point ping-pong at 7.30 us against NCCL's 37.41 us at 20 KiB -- for barlink_host, the host-staged class my verdict assumed was prohibitive. A PP crossing is one-way 10 KiB against a 20 KiB round trip, so that is a generous upper bound, and 29 extra crossings come to 0.212 ms/token, about 0.7% of a ~30 ms round. "31 crossings vs 2, therefore foreclosed by the interconnect" does not survive its own arithmetic even host-staged. So case (b)'s conclusion now stands on ONE ground only: barlink exposes no send/recv, so nothing carries a p2p crossing today. It does not stand on cost. Both original supports -- "no P2P" and "15.5x is prohibitive" -- are withdrawn. scripts/probe/barlink_vs_nccl.py states the API absence outright: barlink "implementiert ausschliesslich Kollektive ... und KEIN send/recv". DESIGN_407:131 MISCITES ITS OWN SOURCE. It credits EVAL_gdr_uebernahme.md:141, a dmabuf-GPU-RDMA-over-RoCE document that contains none of the numbers -- grepping it for 45.59, 1.13, 1.34, interleaved returns zero matches. The real source is FEATURES_VS_UPSTREAM.md:1341 plus commit 137e3a6, which the same row also cites. The measurement is sound, the pointer is not. The same document also claims scripts/p2p_readiness/ has never been run; its results directory exists. Two stale claims, filed as doc corrections. A 2-RANK WEAK SPOT the 3-rank ratios hide: on the fast x8 pair the transport loses 1-8 MiB, down to 0.81x (FEATURES_VS_UPSTREAM.md:1349). This does not disturb the case (a) re-pricing -- sgl-project#705's collectives are 3-rank on a TP=3 group -- but BAR1 is not a uniform win and a 2-rank re-pricing must not reuse those numbers. THE P2P TOOLING EXISTS AND HAS NEVER RUN. scripts/probe/p2pproof.cu prints P2PDATA/CANACCESS lines; searching the whole /spinning tree (~30 worktrees) for both literals finds them only inside the source file, never as captured output. The instrument for the one number this analysis lacks is written and unexecuted -- now a cheap window item. Catalog amended same-step with the miscitation and the 2-rank caveat. Every new claim verified at file:line before writing. Desk only, no boots.

… Hebel ist dort schon aktiv und heute schon vermessen. Zwei Boots gespart. Posten 1 des Zahlen-Pivots, am Pin geprueft statt gebootet. == BEFUND == `--rank-tp-ratio` ist ein BOOT-Flag (nicht im stehenden Boot schaltbar) und auf unserer Form strukturell wirkungslos: * Unsere Form ist `tp_size=1, pp_size=3` (aus dem laufenden Boot-Log). Ein expliziter Vektor muss Laenge tp_size haben -- bei tp_size=1 ist das ein Ein-Element-Vektor, also per Konstruktion keine Unebenheit. * `auto-performance` wird unter pp_size>1 ausdruecklich abgelehnt (server_args.py:11846ff). == UND DER GUARD SAGT SELBST, WO DER HEBEL STATTDESSEN LIEGT == Woertlich aus der Ablehnung: "Under PP the distribution of MLP mass across stages IS the layer cut: use --pp-solve-cut or --pp-layer-ratio for that, and --phase-flip-tp-vector for the TP phase's weight split." Genau dieser Knopf ist auf unserer Form AKTIV (`phase_flip_tp_vector=32,16,16`) UND ICH HABE IHN HEUTE BEREITS VERMESSEN (Posten 4 des sgl-project#855-Strangs): 32,16,16 Welt 592.677/533.333 (A/A) Decode-Median 53,5/54,4 (n=2) 34,15,15 Welt 559.395 Decode-Median 56,9 (n=5, Spreizung 6,2%) 42,11,11 Welt 374.755 (-33%) Decode-Median 60,1 (n=5, Spreizung 0,6%) Verdikt damals und unveraendert: 32,16,16 bleibt; 34,15,15 nicht trennbar, 42,11,11 ein schlechter Tausch (+5,6% Decode fuer -33% Welt). == KLEINE INSTRUMENT-DISKREPANZ, benannt == Der Hilfetext des Flags sagt pauschal "Pure tensor parallelism only: combining it with --pp-size > 1 is rejected". Der CODE lehnt nur `auto-performance` ab und erlaubt `auto` sowie einen expliziten Vektor. Wer nur den Hilfetext liest, haelt sgl-project#709 unter PP fuer hart gesperrt; wer nur den Guard liest, haelt es fuer offen. Beides fuehrt in die Irre -- die wahre Antwort ist, dass tp_size=1 den Vektor bedeutungslos macht. == FOLGE FUER DIE PROGNOSE == Die +0,780 ms/Runde aus sgl-project#705 sind fuer einen REINEN TP-Boot gepreist. Auf der Flip-Form ist die aequivalente Groesse der phase-flip-tp-vector, und dessen Messung liegt vor. Der "billigste Decode-Hebel" ist auf dieser Form also bereits gezogen -- nicht ungenutzt. EMPFEHLUNG: sgl-project#709 als "auf der Flip-Form gegenstandslos, auf einer reinen TP-Form offen" im Register fuehren, statt als offenen Hebel. BELEG-STUFE: DESK-BEWIESEN (Flag-Definition, Parser, Guard und tp_size/pp_size aus dem laufenden Boot-Log). Die zitierten Vektor-Zahlen sind BOOT-GEMESSEN vom 2026-08-30, n wie angegeben.

Ying1123 added 3 commits July 23, 2024 20:11

update vllm version

11a4bff

fix context len

1adee18

fix

ff29094

Ying1123 merged commit 444a024 into main Jul 23, 2024

Ying1123 deleted the ying-llama31 branch July 23, 2024 20:49

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Update vllm version to support llama3.1 (sgl-project#705)

4105429

Navigation Menu

Uh oh!

Update vllm version to support llama3.1 - #705

Update vllm version to support llama3.1#705
Ying1123 merged 3 commits into
mainfrom
ying-llama31

Ying1123 commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant