update flashinfer by Ying1123 · Pull Request #135 · sgl-project/sglang

Ying1123 · 2024-02-03T11:10:44Z

No description provided.

Ying1123 · 2024-02-06T03:35:27Z

Close this PR. Flashinfer has been formally released!! We can switch to importing from Pypi.

…ct#135) - Restrict optimisation in PR sgl-project#120 to Llama-3.1-8B due to regression in Llama4 and Llama3 70b Signed-off-by: Rahul Vijayaraghavan <rahul.vijayaraghavan@intel.com>

* fix: resolve DeepSeek V3.1 function calling regression This commit addresses major regression issues in DeepSeek V3.1 function calling: **Issues Fixed:** 1. **Streaming Parser Buffer Management**: Fixed incorrect buffer handling that caused parsing failures in streaming scenarios 2. **Missing Token Filtering**: Added missing `<｜end▁of▁sentence｜>` token filtering that was present in V3 but missing in V3.1 3. **Partial JSON Parsing Logic**: Implemented proper incremental parsing for V3.1's simplified raw JSON format (vs V3's markdown blocks) **Key Changes:** - Enhanced regex patterns with separate handling for complete vs partial tool calls - Improved incremental parsing for V3.1's `{function_name}<｜tool▁sep｜>{json}` format - Better streaming logic with robust error handling and token cleanup - Fixed buffer truncation after complete tool calls **Root Cause:** DeepSeek V3.1 simplified the function calling format from V3's markdown-wrapped JSON to raw JSON, but the detector wasn't properly handling this in streaming scenarios, leading to parsing failures and function calling regression. **Testing:** - ✅ Complete tool call parsing - ✅ Streaming incremental parsing - ✅ Multiple tool calls handling - ✅ Proper JSON validation These fixes should restore DeepSeek V3.1 function calling performance to match or exceed V3 levels. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: resolve tool call parsing logic bug and apply linting This commit addresses a critical parsing logic bug identified by bugbot: **Bug Fixes:** 1. **Eliminated elif condition preventing partial streaming**: Replaced problematic elif logic that blocked argument updates when structure was complete but JSON was incomplete 2. **Fixed buffer cleanup logic**: Changed buffer cleanup to be based on structural completeness rather than JSON validity 3. **Improved function name streaming**: Function names are now sent as soon as the separator token is present, not waiting for structural completion **Key Logic Changes:** - Unified tool call matching with single regex pattern for better streaming support - Separated structural completeness from JSON completeness in completion logic - Enhanced streaming by prioritizing structural markers over JSON validation for flow control - Better buffer management that handles incomplete JSON in complete structures **Technical Details:** - Single regex pattern: `<｜tool▁call▁begin｜>(.*?)(?:<｜tool▁sep｜>(.*?))?(?:<｜tool▁call▁end｜>)?` - Completion criteria: structural completeness OR (JSON complete AND structure complete) - Buffer cleanup: based on finding `<｜tool▁call▁end｜>` marker, not JSON state **Linting:** - Applied Black formatting as required by pre-commit hooks - All code style checks now pass This resolves the duplicate calls and missed updates issues in tool call parsing. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: “Mark <“74402146+TechnoFinch@users.noreply.github.com”> Co-authored-by: Claude <noreply@anthropic.com>

…roject#135)" (sgl-project#158) This reverts commit dfd9b56.

Same rename pattern as session_id_passthrough (sgl-project#135) — proxy + event emission is daemon I/O infrastructure, not a numbered PLAN task. Changes: - git mv verify/t4 -> verify/daemon_proxy_events (preserves blame on the 1116 LoC verify.py + the historical results/ logs) - README banner updated: dropped pre-round-9 NOTE; documented current contract (4 request-path event kinds; in-process daemon-internal pause gate); explicit TODO links to T21 (sglang PUT /aginfer/program_paused) and T39 (proxy disconnect-awareness) for when those land - verify/README.md top-level index: row 5 updated from 't4 / T4 / legacy banner' to 'daemon_proxy_events / (infra) / post-T33 pass' Test run on Qwen3-0.6B-grade STUB sglang (no GPU) post-T33+T20: 12 steps + 4 recovery probes ALL PASS. Proxy overhead p50 = 1.49 ± 0.01 ms over 5 runs × 50 reqs (latency budget unchanged from round-1 audit baseline). Test scope unchanged; only the dir name + README header refreshed. Pending Tn that extend this surface — not yet implemented, so probes deferred: T21 (PLAN §3) — sglang-side PUT /aginfer/program_paused so per_program_usage[p].state is authoritative on sglang T39 (PLAN §4) — proxy gate awaits BOTH the pause condition AND request.is_disconnected(); release with HTTP 499 + transition ENDED on disconnect Webhook-driven event kinds (memory_pressure / PRESSURE_CRITICAL / APPLY_FAILED / HASH_COLLISION / SESSION_END) are covered by old t5 (webhook router); cleanup is sgl-project#137 next.

…rror Renames following the infra non-Tn pattern (sgl-project#135 / sgl-project#136). Also fixes a latent NameError in cold_start_probe introduced by my earlier event_router.py rewrite (commit 119faf5): daemon/event_router.py cold_start_probe: The new schema rewrite kept the synth-payload's used_bytes/cap_bytes fields but dropped the lines that compute them — `used` and `cap` were left as undefined names that would NameError when occ > theta_hi. The t5 stub deliberately reports occ=0.885 to trigger this exact path; was caught when the verify ran post-rename. Fix computes used = sum(subpool.used_bytes), cap = sum(...). verify/daemon_webhook_router/verify.py: - Updated stub /aginfer/state to emit the DESIGN §5 (post-T17) schema shape; pre-fix returned the legacy tier_usage/page_size flat dict which would now KeyError the daemon's startup probe. - Fixed firer.maybe_fire signature drift: old call `maybe_fire(used_tokens=58000, cap_tokens=65536)` used kwargs that no longer exist; production API is `maybe_fire(occ: float)`. Test now passes 58000/65536 ≈ 0.885. - README header banner rewritten; pending probes for T23 / T24 / T31 (APPLY_FAILED / HASH_COLLISION / SESSION_END webhooks) noted. Test result: all 11 layer-A probes PASS. Arrival→handler p50 = 0.80 ± 0.01 ms, p99 = 1.10 ± 0.09 ms (5 runs × 50). verify/README.md top-level index: row 6 updated.

Same infra-rename pattern (sgl-project#135/sgl-project#136/sgl-project#137). No code change in the verify itself — t6 already had no schema-stale refs (the program tracker is a pure FSM; doesn't touch /aginfer/state). README: - Banner rewritten as 'infrastructure regression-guard' status. - OPEN WORK kept (still applies — T33 didn't add ENDED state) but now references T31 (SESSION_END EventKind) + T41 (mark_ended + EventBus.forget) by Tn instead of the deleted N3_GAPS catalog. verify/README.md row 4 updated. Test result: 10 steps PASS in 213 ms. No defensive-fallback / stale- schema findings in this refresh.

…d the draft's guess was wrong The feature inventory is done: 34 features, 17 in each block, in source order with the document's own status tier per row. The useful finding is that FEATURES_VS_UPSTREAM.md already encodes the required order and the redesign should inherit it rather than compose one. Its Block 1 is the heterogeneous-GPU enablers, defined as the features without which three genuinely different GPUs cannot form one usable TP group at all. Its Block 2 is stated as ordered by how much a rig of identical GPUs gains, which is descending normal-rig usefulness already applied. Departing from that sequence is a claim that the source document is wrong and has to be argued. An earlier version of this draft guessed the Block 2 order as speculation first, then scheduling, then KV offload. That was wrong: the document puts the GGUF family first. The error is kept in the file rather than quietly corrected, because it is precisely the failure the ordering rule exists to prevent -- ranking by what sounds impressive instead of by what an ordinary rig gains. On reflection the document is obviously right, since GGUF is what lets someone run a model at all on a card that could not otherwise hold it, while speculation makes an already-working setup faster. Section 7 carries the evidence discipline forward, because a README is where that usually gets lost. Every figure in the source is a lower bound measured on a rig that is unfavourable on every interconnect axis, and that sentence has to survive into the README rather than be dropped to make the numbers look better. The three identities that do not qualify as validation are listed so the README cannot lean on them, and the rule is that every number carries its unit, its hardware and its status tier while a feature without a measurement says so in the place a number would go. Still a draft. Nothing published, no README modified.

…es, and supersede the shadowed tag Closes the draft's own blocking item 2b. Every feature in FEATURES_VS_UPSTREAM.md now carries an evidence label with the figure quoted verbatim: measured when a run produced it, estimate when the document itself says it is not a measurement, absent when there is no defensible number. The tally is 16 measured, 1 estimate, 17 absent -- half the fork has no number, and those rows keep an empty number slot rather than an invented one. Two things the table forces the README to say out loud. Four rows carry figures that are not wins and would be overclaims if presented as gains: the cross-algorithm bandit LOSES to the static winner at 75.52 against 89.22 tok/s, session KV spill lands at 37.74 vs 37.92 ms per verify round which is inside the boot-to-boot band, per-message-class link selection lands inside its own control's spread and the document says it is not worth it on this rig, and both cross-rig PP and the prefill satellite are costs bought deliberately for capacity and for undisturbed decode. SWA-DCP's ~+6-10% is marked estimate and must never be quoted as measured -- the document calls it ex-ante. Every figure carries the lower-bound framing and the detection limit, because without them a number from this rig reads as a benchmark: raw tok/s has a 2.6-4.2% boot-to-boot spread against 0.09-0.85% for ms per verify round, so anything finer than ~3.5% belongs on the round-time axis and values inside the spread are not gains. The summary line above the table first said fourteen absent rows. It was seventeen. The count is now taken mechanically and the error is recorded in place, since a miscounted summary sitting above a correct table is exactly how an invented number enters a document written to keep them out. Checklist 7.0 makes superseding the published tag a mandatory pre-push step. The characterisation matters and the loose version is already circulating: the published cu130-nccl2307 is NOT armless. Two distributions provide sgl_kernel and the fork's armed files currently win, so the arm is present today. The defect is instability rather than absence -- pip reports no conflict, so the next install of either one silently decides which files win, and one candidate is armless. An image that is correct today and one pip install from being wrong is not shippable, which is why the guard treats a shadow as an error even with the arm present. The new build must take the tag, the new digest must be recorded, and the sgl-project#416-style verify must re-run against that digest by sha256 rather than by tag, since a tag can move between push and check.

… Layout-Metadatum als Quelle (fnFL2x72/x73) Die Nadel nach dem NF-Flip war seit x52 MISS ('4711' statt '4711-QX', mit langer Antwort 'Code: 100000000'). x69 schloss den QSA-Selektor aus (Budget 8192 = dicht, weiter MISS), x70 das Chunking (D in 4 Chunks MATCH), x69-x71 zeigten D nach dem Flip-Paar auf EIGENEM Prefill kaputt. x72/x73 mit SGLANG_WEG2_SEAM_DIGEST=1 nannten die Stuecke: P nach dem ersten Wake: PP1 k_norm 3, q_norm 3, shared_expert_gate 11; PP2 2/2/8; PP0 nur w13/w2_weight_shape. D nach seinem Wake: TP0 dieselben Klassen NUR in den Layern 29-47 (die Layer von PP1/PP2). Wurzel: Rang n beider Gruppen liegt auf derselben Karte (0 = 5090: D TP0 Attention-Host + P PP0; 1, 2 = 3080er: D-Worker + P PP1/PP2). `_pick_source` bevorzugt den co-located Rang. Ein Form-A-Worker laedt nur Experten und Router (`worker_keeps_parameter`, F3); seine q_norm/k_norm/ shared_expert_gate stehen UNINITIALISIERT im Modulbaum, und `card_inventory` publizierte sie als Quelle. P PP1/PP2 nahmen beim Wake Muell-Normen von den Workern, rechneten damit den Prefill (falscher KV im Store) und gaben den Muell beim Sleep an D TP0 zurueck (Layer 29-47). P startet im Boot schlafend, also lief JEDER P-Prefill auf falschen Normen. Fix: `not_a_source_here(model)` -- EINE Regel fuer Manifest (card_inventory) und Plan (derive_leg_plan): auf einem Worker fallen die von `worker_keeps_parameter` vetoierten Namen ("form-a-worker-unloaded"), auf jedem Rang die `*weight_shape`-Tensoren ("layout-metadata": lokale logische Form, nur beim Laden gelesen; x72: 29+29 je P-Rang mit dem Digest des anderen Layouts). Abwesend aus dem Manifest plant der Join sie auf den Raengen, die sie halten (die meta-no-bytes-Form); PP1/PP2 beziehen die Normen dann vom Attention-Host ueber die Karte. Tests: test_weg2_form_a_worker_manifest_x72 (4: Worker-Inventar, klassischer Rang, Praedikat = Ladepraedikat, beide Walks fragen dieselbe Regel); test_weg2_seam_digest_1350 Fixture um named_modules() ergaenzt (sgl-project#135 liest es in expert_buffer_tensors; der Smoke-Test lief seitdem UNARMED). 68 passed.

… -- front-Deadman lebte nach dem Teardown weiter fnFL2h91v1 (26.09.): der vom Launcher gestartete front-Deadman (GRACE_S=600, das front-Log traegt keinen READY-Marker) lief nach dem Arm-Teardown (08:00:50Z) bis zu seinem Deckel weiter, Verdikt erst 08:05:58Z -- der Arm kannte seine PID nie (nur das Zustands-JSON, das nur --teardown PATH liest). Fix: register_helper schreibt fuer die memts-Aufsicht und die drei Deadmen (alle Sitzungsfuehrer, pgid == pid) je einen Eintrag {EVIDENCE_DIR}/helpers_<tag>/<name>.pid = 'PGID START OWNER OWNER_START NAME' (START = /proc stat Feld 22, OWNER = der Arm als Elternprozess); boot_helpers.sh im Arm liest dasselbe Format und beendet die Gruppen im Teardown bzw. per Eigentuemer-Waechter, wenn der Arm ohne Teardown stirbt. teardown() TERMt Deadmen ueber die Gruppe (nur bei passendem argv, sonst wie bisher die PID) und loescht die Eintraege. Schreibfehler am Register kosten nur den Gruppen-Stop, nie den Boot (Zeile 'WEG2-HELPER NOT registered'). Tests: test_weg2_helper_registry_h135 (neu, 7); H52-Suite schreibt ihr Register jetzt ins tmp (EVIDENCE_DIR gepatcht). Noch NICHT gelaufen (pytest gesperrt bis 'Tests frei'), nur py_compile.

…ung -- Deckung ueber alle Tags + Riegel fuer ungedeckte Tensoren Prior-Art statt zweiter Mechanik: D ohne Platte gibt es seit sgl-project#108/sgl-project#109 (22.09., --weg2-d-adopt on: D laedt --load-format dummy, W115 sperrt jeden Forward, der Erstflip bringt Dense UND Experten-Praefix ueber die Legs -- die Platztausch-Puffer sgl-project#135 stehen in Manifest/Plan/Coverage --, kalte Zeilen kommen aus dem Store sgl-project#109). Am Metall zuletzt w87 (22.09.): 0 Presplit-Zeilen auf D. Seitdem sammelt der Wake je TAG (sgl-project#1374), und genau dort war die Naht offen: * _weg2_wake_reload_weights kehrte im per-Tag-Zweig ZURUECK, bevor der sgl-project#108-Block lief -- mark_adopted wurde nie gerufen, W115 fiel nie. * _weg2_last_inject_cover hielt nur den LETZTEN Tag (filled eines Tags gegen expected des ganzen Plans) -- selbst erreicht haette die Deckung nie gereicht. * Die Deckung zaehlte nur Plan-Deskriptoren: ein Tensor ohne Deskriptor (Peer haelt ihn nicht) behielte still seine Dummy-Bytes, waehrend der Riegel faellt. Fix: _weg2_adopt_accumulate sammelt je Tag (filled je Tag, expected = ganzer Plan, geschriebene Namen ohne ZEROFILL), _weg2_adopt_settle laeuft in BEIDEN Wake-Formen, _weg2_adopt_uncovered vergleicht jeden Tensor des Rangs (_weg2_rank_param_table) mit den geschriebenen Namen; SGLANG_WEG2_D_ADOPT_UNCOVERED=refuse (Default unter Adoption) laesst den Riegel dann stehen, =log nennt nur. Ohne --weg2-d-adopt ist alles inert (accumulate/settle tun nichts, solange keine Platzhalter stehen). Marker: 'sgl-project#108 ADOPT-COVER filled=N expected=N uncovered=0 -> PLATZHALTER GELOEST'; rot: 'RIEGEL BLEIBT', 'sgl-project#108 ADOPT-UNCOVERED n=.. first=..'. Test test_adopt_per_tag_bootzeit3_0929 (7): auf 4e153b8 rot (API fehlt), gruen mit Fix; Nachbarn adopt_erstboot/repack_adoption/store_als_quelle gruen. weg2-Wake-Suiten (flip_legs_h11, xchg_* u. a.) sammeln am Desk nicht ('Invalid device id', identisch auf der Basis).

…ary_only / 0 missing），TP2 计时协议缺陷如实列 unverified；最大风险=/tmp per-sample 资产易失，P0 建议零 GPU 归档入库 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…after the GGUF door; host-peak line names the whole owned set Review 1 (major): after the GGUF door the expert parameters w13_qweight / w2_qweight still held the whole [R+C(+X)] bank, unmarked, over the same storage as the published weg2_experts_* buffer. card_inventory therefore published the bank twice (parameter + sgl-project#135 buffer) until MoEExpertOffloadCache.install marked the alias on the first forward: the W84 / W68 class. Fix: presplit_gguf_param swaps param.data for a 0-row placeholder in place after publication, as the Marlin door does (expert_offload.py p.data = empty); install builds the real Parameter from _moe_offload_presplit anyway. Review 2 (major): the streaming-off log claimed "host peak = this layer's loaded experts". Wrong: materialize_gguf_weights runs from process_weights_after_loading after the COMPLETE load pass, so the whole owned expert set (all layers) is in host anon memory first. The line now says so, once per process, and names the per-rank size from the GGUF header (owned_expert_host_bytes: sum over layers of owned experts x that layer's row, never a mean row); without a readable header it says "size not derived from the header". No behaviour change (streaming stays off under a store / Karte / seat rows). Tests (via pytest_gedeckelt.sh, CUDA_VISIBLE_DEVICES empty, one invocation of 24 files): 298 passed, 0 failed, 0 skipped, 3 subtests passed. weg2/test_gguf_g4_platztausch_store_1009.py 63 passed (8 new: placeholder + real card_inventory over staged P / D0 / D1 layers, post-install inventory, seat-row bank, owned_expert_host_bytes, host-peak line with and without header) model_loader/test_gguf_host_residency_644.py 6, layers/test_expert_offload_repack_door_421.py 8, layers/moe/test_d_store_adopt_bootzeit_h2.py 10, test_d_store_identity_h2c.py 13, test_store_prefetch_bootzeit5c_0929.py 8, test_repack_skip_vetoed_bootzeit3_0929.py 5, test_store_write_runs_bootzeit3_0929.py 4, test_streaming_stager_race.py 6, test_cold_tier_wiring_394.py 6, test_lazy_expert_staging_396.py 18, weg2/test_weg2_nvfp4_offload_half_h68b.py 20, test_weg2_nvfp4_ct_unchanged_h68b.py 6, test_weg2_platztausch_residency_x100.py 9, test_weg2_platztausch_d_load_path_rc12z29c_239.py 8, test_weg2_platztausch_owned_vector_rc12z29c_239.py 5, model_loader/test_h2d_read_rows_bootzeit5d_0929.py 9, test_presplit_reclaim_bootzeit4_0929.py 5, weg2/test_weg2_gguf_flat_declaration_g1.py 8, model_loader/test_gguf_qwen4exp_g1_1009.py 23, weg2/test_weg2_gguf_launcher_g2_0925.py 33, test_weg2_gguf_weights_g6_0925.py 11, model_loader/test_gguf_dequant_workspace_chunk_f1b_0925.py 6, test_gguf_postload_chunk_scope_0925.py 8 Not run: test_planer_referenz_n3_1006 (TestDryRunGolden, 4 red on an untouched c651892, host evidence); no GPU, no boot.

…-20261014 [Speculative] Allow DFlash with FlashInfer MegaMoE under attention DP

Ying1123 added 2 commits February 3, 2024 11:10

update flashinfer

204bbe7

update flashinfer

7f8b8ac

merrymercy force-pushed the main branch 2 times, most recently from 1f41598 to 8ff870b Compare February 5, 2024 11:29

Ying1123 closed this Feb 6, 2024

Ying1123 deleted the flashinfer branch February 6, 2024 03:35

CSEEduanyu mentioned this pull request Jan 26, 2025

[Bug] NCCL Crash with SIGSEGV Frequently when deploying deepseek v3 #2803

Closed

5 tasks done

NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025

[Visualizer] fix rms_norm json parse error (sgl-project#135)

a4b9dd1

sammysun0711 pushed a commit to sammysun0711/sglang that referenced this pull request Jan 9, 2026

[Script] Update new VLM accuracy test for Qwen3-VL (sgl-project#135)

6d2106d

vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026

Revert "fix: resolve DeepSeek V3.1 function calling regression (sgl-p…

9fe2b8f

…roject#135)" (sgl-project#158) This reverts commit dfd9b56.

Jiminator added a commit that referenced this pull request Oct 16, 2026

Merge pull request #135 from radixark/codex/dflash-flashinfer-megamoe…

f747367

…-20261014 [Speculative] Allow DFlash with FlashInfer MegaMoE under attention DP

Navigation Menu

Repository navigation

update flashinfer - #135

update flashinfer#135
Ying1123 wants to merge 2 commits into
mainfrom
flashinfer

Ying1123 commented Feb 3, 2024

Uh oh!

Ying1123 commented Feb 6, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Ying1123 commented Feb 3, 2024

Uh oh!

Ying1123 commented Feb 6, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant