Update README.md by eltociear · Pull Request #58 · sgl-project/sglang

eltociear · 2024-01-20T03:37:25Z

Langauge -> Language

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

acc fix for moe (sgl-project#58) add moe support for gptoss format refinements minor clean minor fix

* fix: req hang when max_new_token is 1 and change resource repo * fix: change ci workflow code path * change test ci args

…'s verdict table, the stage runtime Slices 4, 5 and 6, plus the two riders the first metal test needs. weg2/vision_stage_runtime.py (slice 6) -- the order the pieces run in, and the guarantee that the card is left as it was found on EVERY path out, including the ones that raise: whatever was displaced is restored, and whatever was loaded is released, before the stage returns OR raises. That invariant is the module. A stranded tower is 0.858 GiB the prefill will not find -- an OOM, loud. A stranded weights band is a HOLE in the weights: the next forward reads garbage and returns plausible wrong text, quiet. So the restores run BEFORE a release failure is escalated, and VisionStageTeardownIncomplete is deliberately NOT a VisionStageRefused -- a caller turning refusals into 501s must not swallow a compromised boot. Every side effect is an injected hook, so the whole stage including every failure path runs at a desk with no GPU, no NVML and no checkpoint. weg2/launcher.py (slice 4) -- `--weg2-vision transient`. It does NOT pass --no-enable-multimodal: that flag also switches off the tokenizer's image path (model_config.py:573), so no mm_items are ever built and there is nothing for a stage to encode. It sets the hf-config override language_model_only instead, which vision_tower_forced_off (qwen3_vl.py:1230) honours just as well while leaving the tokenizer alone. vision_model_flags() is pure, so the three forms are enumerable at a desk. weg2/front.py (slice 5) -- _video_parts and vision_verdict. The video hole was real: _image_parts counted three IMAGE type names, so a video part scored 0, routed as text, and reached _require_visual on a tower-less rank -- past the refusal that was supposed to be structural. Video now refuses by name (W103) in EVERY mode including transient, because a clip's encoder cost is quadratic in the patch rows and its frame count is unmeasured here. An unknown mode refuses too (W104): `route` is the one verdict that can return wrong text, so it must never be reached by falling through. models/qwen3_vl.py -- weight_name_needed. The loader looks it up by name (loader.py:763); this class defined none, so every text-only rank read the checkpoint's tower off disk and dropped it by name afterwards: 0.858 GiB a rank, six ranks, ~5.15 GiB per boot. Narrow on purpose -- it vetoes the tower and only when self.visual is None, so a boot that builds one has a byte-for-byte unchanged load path. model_executor/vram_family_census.py -- [vram-idle]. [vram-peak] is the right instrument for a pool that coexists with the prefill and the WRONG one for a stage that runs before it and is gone before it; sizing against it understates the card by the whole transient (1.69-2.47 GiB on fn8aj). Emitted at "after pools", the one moment the reading is honest. NAMED GAP, not faked: the group-side call of run_vision_stage is not wired -- it needs the census, the memory-saver tags and the checkpoint, none of which the front process has. So `transient` refuses an image today (W102, 501) and the message says exactly what is missing. Routing it would reach _require_visual on a tower-less rank, which is wrong text, not an error. Named in the code, in the refusal, and in a test. Tests: 55 new (18 whole-stage probe, 37 wiring), 113 vision tests total, all hermetic. Gate: pytest -q tests/moe_offload -> 43 failed, 756 passed, 41 skipped -- the 43 are the recorded baseline and none of them is a vision test.

… RUNS instead of refusing Slice 9. Round 2 left one gap, named in three places: the stage was callable and nothing called it. This closes it, and the only real decision is WHICH PROCESS runs the tower. Not a rank, and the reason is delivery. Under PP3 only stage 0 turns tokens into embeddings, so a tower run on rank 2's card would need a new cross-rank transport for 10 MiB in exactly the window the layout is supposed to be quiet. The stage runs in the process that already holds the pixels: the P group's multimodal-processor process. It built the mm_items; upstream itself fills precomputed_embeddings there (moss_vl.py:587, wrapped for transport at base_processor.py:1556-1563); it can bind any card rather than a rank's; and delivery is free because attach_precomputed_embeddings refuses a tensor still on the stage's card, so the rows are on the host by construction and _get_precomputed_embedding moves them to whoever asks (mm_utils.py:459-467). weg2/vision_stage_service.py -- the caller. census_from_nvml turns registry.nvml.memory_snapshot() into the planner's input, using the driver's own free_mib (the total-minus-used reading cost boot weg2rg6 424-518 MiB of carve-out reported as free) and DROPPING a card with no measured H2D rate rather than assuming one. Every seam becomes a VisionStageOutcome with a W-code -- W105 no room, W106 load, W107 encode, W108 flip, W109 timeout, W110 teardown -- so nothing escapes as a bare exception, which downstream is a 500 where a named 501 belongs. code_for() is total over the seams for that reason. fatal=True is W110 and only W110: the group is compromised, not the request. The limit of this process is named, not hidden: pause_tag/resume_tag live in the RANK processes, so eviction is forbidden here and a placement that would need a band refuses with numbers and says why. No-eviction placements work end to end today. The timeout is a deadline BETWEEN legs. run_vision_stage grew an explicit check_deadline(leg) hook, called before each leg and never before the teardown -- an interrupted stage can strand a band, which is a hole in the weights and worse than any slow request. (This replaces a first attempt that hijacked the clock callable: it worked but depended on how many times the runtime happened to call the clock, which is not a contract.) base_processor.py -- the one seam, before the transport wrap, inside a try: the text path must not be breakable by a vision seam. A no-op with no service installed, so every other boot is byte-for-byte what it was. front.py -- transient now ROUTES. W102 becomes an info line, and an image request is forced to the P leg (route "long"), because the rows are attached in P's processor and do not exist on D's side at all. "none" is not overridden: a request that fits nowhere still fits nowhere. The round-2 gap test is SUPERSEDED by its negation rather than deleted, so the history shows the change instead of a test quietly vanishing. Tests: 36 new (35 whole-path probe + the superseding one), 150 vision tests total, all hermetic. Gate: pytest -q tests/moe_offload -> 43 failed, 791 passed, 41 skipped -- the 43 are the recorded baseline and none of them is a vision test.

…ient stops being a no-op Slice 9 built the service and the processor seam. Nothing ever called `install`, so a boot with `--weg2-vision transient` was byte-for-byte a boot without it: the seam a permanent no-op, the items leaving with `feature` and no rows, and the failure surfacing three hops later inside a rank as `_require_visual`. This slice is the missing call, as a FIXED part of the boot. weg2/vision_stage_boot.py (new) `arm_transient_vision`, called once from the group's tokenizer process. Builds the design's §6 hook table: nvml_snapshot, h2d_gbps {0:14.4, 1:6.5, 2:13.3} (measured, RANG-LINK-ZUORDNUNG), tower (find_tower_shard -> tower_extent -> tower_from_span), load_tower (tower_state_dict + strip_checkpoint_prefix + module on cuda:<plan.card>), encode (tower forward, rows to the HOST), release_tower, read_gbps 1.08 (BUFFERED, not the unbuilt O_DIRECT 3.85), pause_tag/resume_tag None. `ctx_bytes` is MEASURED, not estimated: NVML free before/after `torch.cuda.init()` in a CHILD process that then dies. In-process would strand a context on a card the placement may not pick, and the post would then be paid twice. The child is pinned by NVML UUID (CUDA_VISIBLE_DEVICES=GPU-...), never by ordinal, and `torch_index_for_nvml_card` is the one place torch's enumeration and NVML's are joined -- by UUID, refusing rather than assuming. Four posts, all booked explicitly, none rounded up (no reserves): weight bytes from the header, ctx from the probe, activation for ONE encoder block (not the sum over depth) at the named acceptance geometry 1024x1024, embeddings = 10 MiB exactly. EVERY missing precondition refuses BY NAME and is never a silent no-op: W111 Weg2VisionArmRefused at boot -- names what was missing W112 Weg2VisionNotArmed at request -- quotes the boot reason `maybe_run` raises W112 when there are items to stage and this boot asked for transient without arming; `base_processor` swallows every other seam failure and deliberately not that one. The arming sits OUTSIDE the `is_multimodal` branch so that transient + `--no-enable-multimodal` (no mm_items ever) is a named refusal too, not an invisible one. The launcher publishes SGLANG_WEG2_VISION for group P and the transient form ONLY, and POPS it otherwise -- launcher output, never operator input (R19), so an inherited shell value cannot arm a stage nobody asked for and a D rank never spawns a context probe. Text path unchanged without the flag: same argv per vision form, no variable published, `arm_transient_vision` returns before touching any state (no NVML read, no probe), `maybe_run` exits before the refusal for items with nothing to stage. W103 (video) untouched in every mode. Tests: tests/moe_offload/test_vision_stage_boot_0920.py, 55 passed, hermetic (CUDA_VISIBLE_DEVICES="", synthetic safetensors header, injected NVML/probe/module). Vision suite total 205 passed. Gate `pytest -q tests/moe_offload`: 43 failed, 846 passed, 41 skipped against a measured baseline of 43 failed, 791 passed, 41 skipped -- same 43, no vision test among them.

… config, one load for all pending images, named verdicts User design 2026-09-24 (binding): the tower runs INSIDE the P group's PP0 rank process, its parameters as views on a tail slot range of the KV pool, read from disk with O_DIRECT, after the wake and before the admission; no second CUDA context, no displacement, no host-RAM post, no VRAM booking. V1 (31f8ba2) built the model-neutral core; this commit is the runner the scheduler calls (V2b wires it, V2c retires the tokenizer-process stage). Standalone: new module + tests, depends only on V1 and on Task sgl-project#58 modules that the 27B line (829ebd0) carries. weg2/vision_rank_runner.py - arm_rank_stage: transient boot, group P, pp_rank 0 only. TP>1 on PP0, a missing vision_config and SGLANG_VIT_ENABLE_CUDA_GRAPH are W111 refusals that still arm the PASS, so images are aborted by name (W112) instead of reaching _require_visual. - build_tower_meta: Qwen3VLMoeVisionModel from hf_config.vision_config, every parameter on meta in bf16 (the loader's dtype; the old stage built fp32 and converted), attention backend from the card's smem ceiling (xsn407), held again for the encode because the forward reads the global. - run_rank_stage: tail pages reserved whole (else W105), parameters placed on the tail segments, checkpoint read into the views (O_DIRECT, 2x32 MiB pinned bounce, host cache emptied), ONE image at a time encoded (activation peak = one image), rows to the host as plain tensors and attached. Teardown on every path: views/module dropped, the rope entries the build added popped from _ROPE_DICT, gc, scheduler-stream sync, tail back at the END of the free list, empty_cache. Never raises; the verdict (W102/W105/W106/W107) carries legs_ms, the NVML per-process residue (W110 above 2 MiB, card found by UUID) and the host memory.current / RssAnon deltas. - vision_rank_pass: stages ALL pending images with ONE load when PP0 is idle (running batch empty, no chunked request, _pp_microbatches_drained); while PP0 drains, EVERY waiting request is held out of the pass (no image starvation, no unstaged forward); refused rids stay held until their abort lands and are never restaged; an exception outside the stage's own verdict is a named W106 abort, never a dead group. - take_origin_aborts: the refusals as AbortReqs for PP0's intake (the request origin), the W-code as the client's finish reason. vision_rank_stage.params_on_meta: the patch acts on the building thread only. Desk gate against the real 27B checkpoint header (Qwen3.8-27B-INT8-gdncov-vocabembed): 333 tower tensors, all BF16, one shard (model-00001-of-00018), prefix model.visual, 0.858 GiB -- the bf16 meta-built module pairs dtype-exact with them. Tests (hermetic, CPU, each file alone in the incg cap): test/registered/unit/weg2/test_weg2_vision_rank_runner.py 20 passed test/registered/unit/weg2/test_weg2_vision_rank_stage_core.py 12 passed

…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153) Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only. - pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted the admission-decision arc the whole file drove (_pp_send/_recv_admission_ decision, _pp_drain_voided_proxy, ...) and made the proxy channel unconditional; #1015c (206661a) retired the downstream refusal (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream pass refused, 0 served). The file now pins both halves of that retirement (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name). - pp_upstream_void_before_formation_951 x3 (found in the same family): the three refusal cases rewritten to the #1015c form (pass reaches the formation, predicate False on every shape, no void -> streak 0, slot kept). - scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency + SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the P->D flip D legitimately admits into; it now models the D->P flip as flip() runs it (awake stays 'D' until the end). Mutant (post-acquire re-check removed) -> red. t3 is the front defect, next commit. - sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst verdict into a local tested in the loop (b1d: locals assigned from a call/attribute count as state; b1e-g: the slice gets its stock values _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time` (real clock). Round-5 mutant (`if req is not None: continue` above the gate) still red on all four. - no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49 epoch keyword of price_remainder (196f6a8), H84 x_start_tokens (b895928). - s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request around _handle_generate_request_impl; the seam is the impl's first statement, the wrapper is pinned to clock-read + delegation. - pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants the usable-match floor on tree_cache; harness tree_cache=None.

…prefilled on P nor admitted to D Port of d04fad1 (desk/nf-h102-unified-0926): per-request watcher, abort per state (queued -> dequeued, p-leg1 -> /abort_request on P, P-slot wait -> leg1-skipped, handoff -> dropped-before-d, parked -> abort on D, d -> leg-2 path), one line WEG2-CLIENT-GONE, switch SGLANG_WEG2_ENABLE_CLIENT_GONE_ABORT (default on). Conflicts resolved with both sides: environ (the new switch beside the profile rows), the CARRIER site (R28 _dp_mark + watcher), the BATCH/SK site (the kept-SHORT D line + R28 + watcher), _on_leg1_done (client_gone before the RVP finish). On the current front, beyond the port: - removal BY IDENTITY (queue loop, _ready_for_d_drop): deque.remove compares Pending field by field (dataclass __eq__). - leg 2 entry: a client that already left is not admitted to D (a SHORT at its gate/seat, a hand-off inside the watcher's tick) -- 499, seat released at once, WEG2-CLIENT-GONE state=pre-d action=not-admitted, counter client_gone_pre_d. - client_gone/response_complete tolerate request doubles (no transport/writer = not gone). Tests: test_weg2_client_gone_h102.py 12 passed (the 9 H102 metal-shape tests with real aiohttp fake P/D, + pre-d not admitted with the seat back, live client / switch off reach D, helpers on doubles).

Update README.md

4193d1f

Langauge -> Language

merrymercy merged commit 0b2efc2 into sgl-project:main Jan 20, 2024

CSEEduanyu mentioned this pull request Jan 26, 2025

[Bug] NCCL Crash with SIGSEGV Frequently when deploying deepseek v3 #2803

Closed

5 tasks done

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Update README.md (sgl-project#58)

f694bc3

yanbing-j pushed a commit to yanbing-j/sglang that referenced this pull request Apr 16, 2025

fix moe fp8 avx512-bf16 (sgl-project#58)

f682d6d

chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request May 13, 2025

fix moe fp8 avx512-bf16 (sgl-project#58)

4494c23

chunyuan-w added a commit to chunyuan-w/sglang that referenced this pull request May 16, 2025

fix moe fp8 avx512-bf16 (sgl-project#58)

8ac14da

blzheng pushed a commit to blzheng/sglang that referenced this pull request Aug 12, 2025

acc fix for moe (sgl-project#58)

8e7969d

gaolaobao mentioned this pull request Aug 25, 2025

[Bug] RTX 5060: RMSNorm failed, same as the #7249 issue, when running qwen2.5-0.5b-instruct model. #9600

Closed

3 of 5 tasks

key4ng pushed a commit to key4ng/sglang that referenced this pull request Nov 9, 2025

bugfix: Fix TextSampler sampling logic (sgl-project#58)

36a4d25

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

blzheng pushed a commit to blzheng/sglang that referenced this pull request Dec 26, 2025

add moe gptoss bf16

7bca989

acc fix for moe (sgl-project#58) add moe support for gptoss format refinements minor clean minor fix

Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026

Fix and Refactoring CI testing (sgl-project#58)

17e003b

* fix: req hang when max_new_token is 1 and change resource repo * fix: change ci workflow code path * change test ci args

vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026

ci: update tgl ci (sgl-project#58)

a3711e7

amd-bot mentioned this pull request May 3, 2026

[CI Monitor] Daily Report - 2026-05-03 bingxche/sglang-ci-bot#58

Open

Navigation Menu

Repository navigation

Update README.md - #58

Update README.md#58
merrymercy merged 1 commit into
sgl-project:mainfrom
eltociear:patch-1

eltociear commented Jan 20, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

eltociear commented Jan 20, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants