Repository navigation
Conversation
|
How much minimum GPU mem is required for loading this model? I tried on 2xH100 with the following command, but i am running into OOM CUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server --model-path databricks/dbrx-instruct --port 30000 --tp-size 2 --random-seed 42 --trust-remote-code |
Collaborator
Author
|
@benihime91 It's a 134B model and the model weight is 246GB. So it takes at least 4 * 80G GPUs to load this. |
|
Ahhh 😢 should've tried with 4xH100 |
Ying1123
approved these changes
Mar 28, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
shiyu7
pushed a commit
to shiyu7/sglang
that referenced
this pull request
Dec 12, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
shun8686
pushed a commit
to shun8686/sglang
that referenced
this pull request
Apr 13, 2026
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jul 14, 2026
…ctable-aware mamba admission gate (sgl-project#338) Two root-cause fixes so the recurrent-bound swarm runs correctly under HiMA: sgl-project#337 (mamba_actuator): MambaArenaActuator.unmark_token_slots routed the k2m dst-grow through the legacy MambaPool.unmark_slots shadow state, which live_size no longer reads, so a grown mamba pool was invisible to BudgetAgent._maybe_update_admission_cap ("grew events: 0", cap pinned at boot). Route through the allocator (self.allocator.unmark_pages_capped -> CappedFreeList._grow_into_tail) and reconcile pool.size from pool.live_size, byte-symmetric with KVArenaActuator. Single source of truth = the allocator's CappedFreeList. sgl-project#338 (memory_pool + common + scheduler): a hybrid request needs a mamba active slot in addition to a req slot, but Scheduler.get_num_allocatable_reqs gated only on req_to_token_pool.available_size() (req slots). After sgl-project#337 raises the cap, the scheduler admitted a batch mamba could not back and alloc_req_slots RAISED instead of leaving the request queued (the design defer action) -> swarm crash. Add HybridReqToTokenPool.mamba_admittable_reqs(mamba_evictable) = (free + evictable) // slots-per-req and gate get_num_allocatable_reqs on it, mirroring the KV available+evictable gate in PrefillAdder. Counting evictable (what alloc_req_slots reclaims via pre-evict) is what keeps a lightly-loaded pool from throttling on free slots alone. per-req factor extracted to HybridReqToTokenPool.mamba_slots_per_req (single source of truth shared with alloc_req_slots). Validated: 2 reproducing unit tests (actuator grow -> live_size; admittable-reqs crash-guard + no-throttle). E2E swarm A/B (real CC subagent-flattened trace, default config, conc 400): both arms complete clean 1200/1200, 0 crash (was 2148 err); base unaffected (1009 tps, ran at 194); sys grows mamba 977->1233, admits to 246, TTFT -23% mean / -30% p99. (Throughput is GPU-compute-bound at real-CC 16K-context, so the higher concurrency converts to a TTFT win, not a throughput win.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 4, 2026
…ections that move the plan SLICE 1, BUILT. models/qwen3_tts_fast_predictor.py replaces the single call that the precursor measured at 104.44 ms of a 142.78 ms frame -- 73 % of it -- with a raw 15-step loop. It is a drop-in: the reference reads .sequences off code_predictor.generate (modeling_qwen3_tts.py:1671) and gets the same shape, so the 1596-line prompt builder, the trunk loop and the codec are untouched. install()/uninstall() keep the reference reachable in the same process, which the identity gate needs and an operator needs as a way out that is not a restart. It holds no weights, so it costs no VRAM. Sampled tokens never leave the device -- every .item() there is one of the 602 syncs per frame this removes, and a host round trip would also make the loop uncapturable in slice 2. The step schedule is data, not a comment: prefill takes lm_head[0] and no codebook embedding, decode step g takes codec_embedding[g-1] with lm_head[g]. Off-by-one there does not crash, it picks a neighbouring codebook and degrades timbre, so step_schedule() is pinned by five assertions. The sampler is the main correctness surface and is driven against transformers' OWN TemperatureLogitsWarper / TopKLogitsWarper / TopPLogitsWarper on fixed logits, mask and values separately, including the production defaults (top_k=50, top_p=1.0, temperature=0.9) and all three no-op cases -- a skip transformers takes and we do not is a silent divergence on a DEFAULT config. One test was wrong and is corrected rather than deleted: temperature and top_k genuinely COMMUTE (monotonic scaling cannot change which tokens top-k keeps), so an ordering bug between those two is unobservable; the detectability arm now uses top_p, which does not commute with temperature. Both facts are pinned. 14 hermetic, 215 green across the models suite. End-to-end greedy code-token identity needs the real checkpoint and the tenant's import shims, so it runs with the GPU measurement rather than here. The gap is named in the test file, not only in this message. CORRECTION 1 -- the 127 MiB is a rebooking and fp8 is NOT the pool gate. My "fp8 is the gate" is withdrawn. But two named levers do not reach this boot, read at source: sgl-project#330's dial needs --enable-vram-dial, which is a BOOT flag the deployed launch line does not carry (/proc/3953200/cmdline), so "no restart" does not apply to this instance; and it additionally requires weighted uneven DCP (vram_dial.py:1110) while the boot has no --rank-kv-ratio at all. The knob that DOES reach it is already in the launch line: --rank-auto-reserve-mib 13000,3800,3800, whose two 3080 entries go 3800 -> 4200 for ~400 MiB each at the next restart, with no new flag and no dial precondition. Priced honestly: 16 full-attention layers x 4 kv heads x 256 x 2 x 1 B fp8 = 32 KiB/token, so 400 MiB costs 12,800 tokens on that rank, ~4.9 % of one 262144 context. bf16 instances then fit and the pool does not wait for fp8. Still to check before any quantisation effort: an sm86 bf16 engine may already fit under 2551 MiB by itself, which is a measurement on the built engine, not a prediction. CORRECTION 2 -- TRT is cut as fork lane infrastructure (sgl-project#337), not a talker side path. The discipline I specified already EXISTS in video_enhance/engine_cache.py: EngineKey over nvml_uuid|device|driver|trt| onnx_sha256|precision|shape triplet|flags, provenance manifests, and "no manifest means absent, not usable". So the plan is promote, not duplicate: lift it to srt/trt_lane/ with video_enhance as its first existing consumer and byte-identical video behaviour as the acceptance condition, then add the sgl-project#337 rung registry, sgl-project#286 ledger registration so a loaded engine is parkable like any other VRAM tenant, sgl-project#407 tier placement, per-regime crossover as a reported output, and sm86 refusals by name. The talker's two engines are the first production consumer; RIFE/SR (sgl-project#469) attach afterwards. Ordering unchanged: graphs first -- the 6.53x needs none of this.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
… chain in one engine The fold is now a first-class arm, not a footnote. Folding w_q x w_scale into bf16/fp16 weights at BUILD time removes the constraint that shaped the INT8 design: with no runtime dequantize there is no activation quantization either, so TensorRT's inability to express a per-token dynamic scale stops mattering. The INT8 arms are per-stage engine islands with our quant kernel between them; folded, chain_decode_layer is ONE engine containing four GEMMs, the SiLU-gate multiply and the attention-core bridge, with nothing of ours inside it. 22 more engines built at the desk, CPU-only AOT, real checkpoint weights, 19 s for all 32. Variants: int8 (per-stage), fold_bf16, fold_fp16 (whole chain), plus one fp32_ref shape per rank as a labelled diagnostic. QUALITY, MEASURED RATHER THAN ASSERTED. The fold does not coarsen anything -- w_q x w_scale is the checkpoint's weight exactly, in a wider container -- and it removes the per-token activation quantization error the deployed path pays. So the harness computes an exact fp32 reference from the dequantized weights and gates each fold on being AT LEAST AS ACCURATE AS WHAT WE DEPLOY, rather than on a fixed tolerance. Gating a fold against the deployed OUTPUT would have measured the deployed path's own quantization error and called it a fold defect. Mock, random weights, directional: deployed 0.0112, fold bf16 0.0031, fold fp16 0.0024, max-abs vs exact. THE MOCK FOUND AN fp16 LIMIT. fp16 wins on mantissa (11 bits vs bf16's 8) and loses on range (5 exponent bits, max 65504; bf16 carries fp32's range). In the 4-GEMM chain_decode_layer the intermediates compound and the fp16 fold produced a NON-FINITE output where bf16 did not. Random weights exaggerate the magnitudes, so this is a flag for the card run and not a verdict -- but it qualifies the "2^-11 beats the INT8 grid" rationale exactly: the deep chains are where range, not mantissa, decides. Non-finite output is detected per arm and recorded as a result, never averaged into a number. THE STAIR IS THE POINT, and the memory cost is now priced per part. Every fold engine measures 1.999x the INT8 weight bytes; full-model fold does not fit the 3080s and was never the proposal. Per-part cost is in the manifest as weight_bytes_fold / weight_bytes_int8 / memory_multiplier_vs_int8 for every engine, and tabulated in TARGET_SELECTION, so the planner can price residency for any subset of the stair. Hot parts folded, everything else stays INT8 and stays offloadable. CROSSOVER, measured under graphs on both sides: the fold trades zero activation-quant kernels against 2x the weight bytes moved in a memory-bound GEMV. sgl-project#368's 78 % quant share was an EAGER number and graph replay already cuts it to ~11 %, so the fold is buying back a smaller slice than the eager figure suggests. New verdict cells fold_bf16_over_torch_graph, fold_fp16_over_torch_graph and fold_bf16_over_trt_int8 across M = 1,2,4,8; the last one separates "the fold helps" from "the engine helps", which are different wins and a per-part decision needs both. The fp32 arm is one shape, labelled DIAGNOSTIC_fp32_ref_over_torch_graph, and is disqualified as a deployment candidate by memory at 4 bytes per weight. It isolates TensorRT runtime quality from the quantization constraints and nothing else. Assumptions independently checked against the fork and the checkpoint before building: every language-model linear is bias=False with no .bias key in the checkpoint; every module in the target set is INT8-quantized (only lm_head, embed_tokens, in_proj_a/b, visual and mtp are ignored); and silu applies to the FIRST half times the second (activation.py:123-125), which is the epilogue the engines emit. Tests: mock_smoke.sh green with the fold arms in the rotation -- 10 points, 52 arm measurements, tolerance AND quality paths exercised on every point, fp16 chain overflow surfaced as a NOTE and not swallowed. 32 real engines built with checkpoint weights. ruff clean, codespell clean. NOTHING executed on a GPU: the cards were held by another window throughout and every step ran with CUDA_VISIBLE_DEVICES="".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.