Repository navigation
Conversation
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 4, 2026
…ections that move the plan SLICE 1, BUILT. models/qwen3_tts_fast_predictor.py replaces the single call that the precursor measured at 104.44 ms of a 142.78 ms frame -- 73 % of it -- with a raw 15-step loop. It is a drop-in: the reference reads .sequences off code_predictor.generate (modeling_qwen3_tts.py:1671) and gets the same shape, so the 1596-line prompt builder, the trunk loop and the codec are untouched. install()/uninstall() keep the reference reachable in the same process, which the identity gate needs and an operator needs as a way out that is not a restart. It holds no weights, so it costs no VRAM. Sampled tokens never leave the device -- every .item() there is one of the 602 syncs per frame this removes, and a host round trip would also make the loop uncapturable in slice 2. The step schedule is data, not a comment: prefill takes lm_head[0] and no codebook embedding, decode step g takes codec_embedding[g-1] with lm_head[g]. Off-by-one there does not crash, it picks a neighbouring codebook and degrades timbre, so step_schedule() is pinned by five assertions. The sampler is the main correctness surface and is driven against transformers' OWN TemperatureLogitsWarper / TopKLogitsWarper / TopPLogitsWarper on fixed logits, mask and values separately, including the production defaults (top_k=50, top_p=1.0, temperature=0.9) and all three no-op cases -- a skip transformers takes and we do not is a silent divergence on a DEFAULT config. One test was wrong and is corrected rather than deleted: temperature and top_k genuinely COMMUTE (monotonic scaling cannot change which tokens top-k keeps), so an ordering bug between those two is unobservable; the detectability arm now uses top_p, which does not commute with temperature. Both facts are pinned. 14 hermetic, 215 green across the models suite. End-to-end greedy code-token identity needs the real checkpoint and the tenant's import shims, so it runs with the GPU measurement rather than here. The gap is named in the test file, not only in this message. CORRECTION 1 -- the 127 MiB is a rebooking and fp8 is NOT the pool gate. My "fp8 is the gate" is withdrawn. But two named levers do not reach this boot, read at source: sgl-project#330's dial needs --enable-vram-dial, which is a BOOT flag the deployed launch line does not carry (/proc/3953200/cmdline), so "no restart" does not apply to this instance; and it additionally requires weighted uneven DCP (vram_dial.py:1110) while the boot has no --rank-kv-ratio at all. The knob that DOES reach it is already in the launch line: --rank-auto-reserve-mib 13000,3800,3800, whose two 3080 entries go 3800 -> 4200 for ~400 MiB each at the next restart, with no new flag and no dial precondition. Priced honestly: 16 full-attention layers x 4 kv heads x 256 x 2 x 1 B fp8 = 32 KiB/token, so 400 MiB costs 12,800 tokens on that rank, ~4.9 % of one 262144 context. bf16 instances then fit and the pool does not wait for fp8. Still to check before any quantisation effort: an sm86 bf16 engine may already fit under 2551 MiB by itself, which is a measurement on the built engine, not a prediction. CORRECTION 2 -- TRT is cut as fork lane infrastructure (sgl-project#337), not a talker side path. The discipline I specified already EXISTS in video_enhance/engine_cache.py: EngineKey over nvml_uuid|device|driver|trt| onnx_sha256|precision|shape triplet|flags, provenance manifests, and "no manifest means absent, not usable". So the plan is promote, not duplicate: lift it to srt/trt_lane/ with video_enhance as its first existing consumer and byte-identical video behaviour as the acceptance condition, then add the sgl-project#337 rung registry, sgl-project#286 ledger registration so a loaded engine is parkable like any other VRAM tenant, sgl-project#407 tier placement, per-regime crossover as a reported output, and sm86 refusals by name. The talker's two engines are the first production consumer; RIFE/SR (sgl-project#469) attach afterwards. Ordering unchanged: graphs first -- the 6.53x needs none of this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
useless. because the user name and email are not good in the git log.