Make the server random by default by merrymercy · Pull Request #488 · sgl-project/sglang

merrymercy · 2024-05-29T22:32:02Z

Do not set the default value of the global random seed, so the server can be random by default

…be random by default

This reverts commit 80a33ce.

Update full-test-npu.yml

…the vLLM route revoked The GPU window stood the TTS stage up and tested the whole doc-derived contract against a real server. DESIGN §9 predicted that a documentation error would pass every desk test and fail on first contact; it did, in nine distinct places, because the stub was built from the same documentation as the adapter. Partway through, a standing user order struck the serving choice: no vLLM anywhere, ever -- everything serves inside htsglang, because cross-asset spill/offload/eviction arbitration needs one runtime with one ledger and a foreign serving process is invisible to that register. The server was killed, all three cards verified back to 0 MiB. What survives the reversal is the contract itself, observed rather than read. The scripts are corrected here because their previous content was simply wrong, and the corrections carry the evidence in comments for the native bring-up (sgl-project#488) that replaces this path: - vllm-omni does not depend on vllm; it is a plugin, and the core has to be installed explicitly, which also pins torch back down (2.13.0 -> 2.11.0). - `--omni` is a sys.argv sentinel, not an argparse flag: invisible to --help, and omitting it does not fail -- it silently serves a plain LLM whose only symptom is a missing /v1/audio/*. The design doc records the remaining seven deltas, the latency result, and what is still owed. Headline: warm streaming first-audio measured 71 ms on a 3080 against a 150-300 ms budget, RTF 0.16 -- the TTS stage is not the risk. The 36 preset clips remain undone; the render mechanism was proven on one clip minutes before the reversal. Full evidence, corridor rows and artifacts: /spinning/gpu-battery-results/2026-08-03_w3_t0_translator/RESULTS.md

…roject#488) Desk investigation against the runtime, ahead of any Phase 3 building. The blocker is not what it looked like. "The decode loop assumes one token per step" is wrong in both directions: the append side is already N-per-step because speculative decoding put it there, and the KV/position side is strictly one-per-step -- which is exactly right, because sixteen codebook entries are one audio frame at one sequence position, not sixteen positions. The residual codes must never enter the KV sequence. The real blocker is a single line that clears input_embeds on entry to decode, while the talker's next input is a summed embedding vector rather than a token id. The CUDA-graph side already carries embeds, so this is a scheduler unblock rather than a graph rewrite. The speculative stack is the wrong vehicle and is rejected on the record: a draft head is the right shape and the wrong contract -- nothing to verify, no target distribution, no accept length, and spec exists to ADD sequence positions. Smuggling a depth transformer through CustomSpecAlgo would poison the spec metrics, the adaptive-draft ladder and the cross-algo bandit for the rest of the fork's life. The precedent that fits is the dLLM lane: a non-speculative custom decode regime, 935 lines plus wiring, with full_logits as the precedent for a non-standard logits shape and customized_info as a wired-but-unused transport for the extra codes. No host exists. sgl-project#274 is a dual lane within one model's world, sgl-project#305 is design-only, DESIGN_333 M5 is unformulated, and ANALYSE_334 already priced this family HARD/L. Effort against the only complete implementation (5192 lines plus a 2102-line bespoke runner, whose authors also could not host it in a stock decode loop): ~16 days. Seven buys a boot-proven model file, desk parity for the trunk, a harness-driven code predictor and a written hook design -- not streaming decode, graph coverage, the vocoder, or the native speech endpoint. The translator cannot move to the native lane before 08-10, so the ladder's rung B ships. Three traps pre-registered, the sharpest being that the checkpoint writes rope_scaling "interleaved" while the factory reads "mrope_interleaved": passed through, the model loads, runs, emits plausible codec tokens and sounds wrong. Also preserves the vLLM-Omni 0.24.0 reference outside the venv being deleted. It is the only source of truth for the 1596-line prompt builder, the predictor loop and the vocoder, and losing it would have cost more than any other artifact here.

…block to sgl-project#488 Two operator rulings. The translator is published as /translate/ under an existing server_name with the existing certificate, instead of as its own subdomain. That removes both remaining user actions -- no DNS record, no certbot --expand against a certificate that fronts a dozen other services -- and with them the last risk to the user's certificate setup. The host already path-mounts /go2rtc/, /llama/ and /plex/, so the pattern is idiomatic there and /translate/ is collision-free. The client had to learn where it lives: every URL is now built relative to location.pathname, and the manifest's start_url and scope are "./" so the installed PWA scope follows the mount point without the server being told about it. The same file works at the site root and under any prefix, which is what keeps the standalone-subdomain template valid too. Verified through the real front door: health and the PWA serve under the prefix, the manifest carries the relative scope, and the WebSocket handshake returns 101. A diff of the live config against the pre-change backup shows 29 added lines and zero deletions or modifications, so the neighbouring locations are provably untouched. The scheduler unblock (the input_embeds clear in prepare_for_decode) is NOT built. Rung B runs its own generation loop and never enters the scheduler, so a gated patch would have no caller and no falsifier -- unvalidated ballast of exactly the kind the desk-written-never-executed rule exists to prevent. It is recorded as a named sgl-project#488 prerequisite in 11.2 instead.

… the ledger gap it hid The first warm GPU number came back at RTF 24.6 on a 5090 for a 0.6 B talker. Two orders of magnitude from anything the architecture predicts is a symptom, not a limit, so it got opened rather than reported. `probe_stage_timing.py` (new) times the three links of one call with the device drained around each: prompt 5.3s | talker 4.3s | codec decode 85.2s <- 90% of the call The codec hangs off `model.speech_tokenizer`, and `Qwen3TTSTokenizer` is a PLAIN class, not an nn.Module (`inference/qwen3_tts_tokenizer.py:44`). It is therefore not in `model._modules`, `model.to("cuda:0")` never reached it, and it carries its own construction-time device snapshot that its encode/decode use to place inputs. Talker on the card, the entire 12 Hz codec left on the CPU in bfloat16. Nothing failed; bfloat16 convolutions are simply that slow. Same shape as the device fault fixed in the previous commit, one level deeper, which is why the fix is one function: it now finds holders STRUCTURALLY -- a plain attribute owning an nn.Module and caching a device -- rather than by name, so a second such holder cannot be stranded silently the way this one was. And it VERIFIES: after the moves it walks every parameter and buffer and refuses if any is off-target, because "we called .to()" is a success claim and a stranded module is silent by construction. Found while reading that code: `codec_decoder` was registered against `inner.code2wav`, which does not exist on this checkpoint. It resolved to None on every load, logged a warning nobody read, and the largest single audio asset (229 MB) was never in the sgl-project#286 register at all -- a registration with reach zero. Now `speech_tokenizer.model`, and a missing module is a hard failure: an unledgered module is memory the register cannot see, park or evict, and the one-runtime law rests on that register being complete. Falsifiers (hermetic, `meta` for `cuda`): the detached holder is NOT moved by the model's own `.to()` (the bug), is moved after retargeting (the fix), a `.to()` that silently does nothing is caught by the state probe, and an explicit device the model is not on is refused rather than recorded. Tests: 295 hermetic pass under CUDA_VISIBLE_DEVICES=99 (was 291). Measured on the 5090, same command before and after: before: prompt 5.3s talker 4.3s codec 85.2s RTF 24.18 after: prompt 0.02s talker 5.2s codec 0.04s RTF 1.15 codec per frame 1739.6 ms -> 0.7 ms `latency_window.py`, warm, n=10, A-vs-A floor 245 ms: TTS whole utterance 4216 ms median for 3.68 s of audio, RTF 1.146 ASR (faster-whisper large-v3-turbo, int8_float16) 96.9 ms for 3.11 s MT still unmeasured -- no endpoint up in this window. The talker is now 99% of the call at ~90 ms/frame. That is the rung-B limitation DESIGN_466 §12 already names (no CUDA-graph capture over the nested code predictor), not a new defect, and it is what sgl-project#488 exists to remove.

…r instrument that can refute it Three things, all desk-executable, ahead of the GPU window. The TP=3 rejection is now a register row (planner/rejected.py, tts_talker_tensor_parallel) rather than only a paragraph, so the wizard cannot propose what the arithmetic already settled. NOT_DEFAULT, not BLOCKED: the shard is representable and correct, it simply loses -- so it stays offerable with the counter-number attached, unlock line included. Scope is "rig", because the collective term is a rig property while the 192-steps-per-audio-second shape is architectural; the verdict is the two together. The register's own brevity contract caught the first draft (gain/cost <= 120 chars) -- the row was shortened, not the contract widened. The precursor instrument (scripts/dev/488_talker_profile/) decomposes a talker forward into wall / kernel-covered / gap / kernels-per-iteration / syncs, and projects the RTF that would remain if the gap went to zero. It runs against an ALREADY LOADED module inside the tenant process -- no second load, no server boot, ~6 minutes, every loop bounded by both an iteration count and a wall deadline. It is built so it can tell us we are wrong: at >= 70 % kernel coverage it prints PREMISE FALSIFIED and says ANALYSE_488's redirect needs revisiting. And it refuses to testify at all unless two known-answer calibration arms separate by >= 0.40 gap fraction AND the known GPU-bound arm stays under a 0.35 ceiling. The second condition is the one that matters here: the card is currently held by a resident serving instance, and on a saturated box EVERY arm looks overhead-bound -- a separation-only gate would have let contention manufacture the answer we expect. Both refusals are executed as tests, as is the falsified verdict wording. Reach correction, cited rather than assumed: "sgl-project#274 lanes carry multi-instance naturally" does not hold at the code level. dual_group_lane.py:35 says the scheduler holds a list of lanes and "Slice B instantiates exactly one lane"; :16-25 make a lane a full-width shell tree over the SERVING model's own parts, not a slot a different model occupies; server_args.py:9524 requires an explicit --rank-tp-ratio because "the lane shares the plan's rank-0 segment". One talker instance per card is sgl-project#305 shaped, and sgl-project#305 is design-only. What slice 1 can honestly claim is that the model file holds no process-global state -- every geometry decision comes from the parallel context at construction, the same axis sgl-project#274's overlays already substitute -- so N instances stay a host problem rather than a model-file rewrite. Written into ANALYSE_488 §6a. Tests: 25 hermetic across the two sgl-project#488 files (13 lane + 12 instrument), plus 190 green on the register's consumers (wizard, wizard_v2, planner generality, rig coupling). ruff + codespell clean on every touched file. No GPU touched: the arb request is rewritten to the one short in-process arm and the TP=3 arm is withdrawn from it.

… all falsified Slot 1 of the field-report queue, run against the live build 9f60fbb742. The probe captures whole 20 ms wire buffers at `playback.schedule`, together with the clock each was scheduled against, and reads them numerically. seams : max step 0.19052, mean 0.01661 inner : max step 0.25858, p99 0.12994 onset : 1%/10%/50% of local peak at samples 173/316/358, lead-in 320 schedule: 268 calls, 1 re-anchored, 1 started in the past, min lead -20.1s (a) The talker's waveform is NOT the source. The onset rises from near silence to half amplitude over ~185 samples, 11.6 ms -- a natural speech onset, not a step. The very first buffer of a turn starts at exactly 0.0 with a DC offset of 0.0. Nothing here belongs to the sgl-project#488 profile mandate. (b) The framing seams are NOT the source. Buffer boundaries are not the outlier: the largest step across a seam is SMALLER than the largest step inside a buffer, and the mean seam step is an order below the inner p99. (c) Scheduling underrun is NOT the source. Of 268 buffer schedules exactly one started in the past, and it is the benign initialization: `cursor` is 0 while the context clock is at 20.1 s, which is precisely the case the re-anchor exists for. No genuine late start occurred. The rise metric was wrong in the first pass and is fixed here: slicing the stream AT the threshold crossing and then asking how long the rise took reported 0 samples, which was true by construction. The window now starts 320 samples before the crossing. What this leaves, and it is not in the measured set: the client builds one `AudioBufferSource` per 20 ms frame at the WIRE rate -- `createBuffer(1, float32.length, rate)` with rate 16000 -- while the output context runs at 44100 (measured, `playback.diagnostics`) or 48000 on a phone. Every 320-sample frame is therefore resampled INDEPENDENTLY by the browser, with no continuity across frames. This probe reads the input samples and by construction cannot see an artefact created in the resampled output, which is exactly where that one would live. Not fixed here: nothing is faded blind, per the standing order.

…ed by no kernel Ran on the 5090 with coordinator GO, user not testing. The in-process entry point cannot be injected into a live PID, so this used the standalone one: a short-lived second copy beside rank 0 and the tenant. NO tenant restart and no tenant code change -- pid 3954713 verified alive and HTTP 200 afterwards, and the card's free VRAM returned to exactly its pre-run 3605 MiB. The instrument passed its own gate before testifying: the known GPU-bound arm sat at a 0.000 gap fraction against a 0.35 contention ceiling, the known launch-bound arm at 0.851, separation 0.851 against a required 0.40. The GPU-bound reading doubles as lever 5's answer -- the card was not SM-contended during the run. Numbers. Trunk step 38.34 ms wall / 5.64 ms kernel (85.3 % gap). Predictor step 6.07 / 0.98 (83.8 %). The HF generate() envelope 104.44 / 16.21 with 602 SYNCS per frame. Derived frame 142.78 / 21.85 -> RTF 1.713 measured, 0.262 kernel-only, recoverable factor 6.53x. Against the live 5511 ms first-audio that is 38.6 frames, i.e. ~3.2 s of audio: the token loop is essentially the whole TTS latency, and agent 8's tenant-side split will say the rest. Three caveats are in RESULTS.md rather than buried: the event COUNT is inflated because it sums non-leaf profiler events (the TIME uses self-device-time and is not), RTF 1.713 is this configuration and not the tenant's 1.23, and the gpu-bound arm proves "not launch-starved" rather than measuring a ratio. The measurement re-ranks two things. TRT gains a numeric argument rather than deference: kernel time is 21.85 ms against a 2.2 ms bandwidth floor, a 10x that is kernel SHAPE, so graphs remove the 84.7 % gap and only fusion touches the rest -- ladder 1.71 -> ~0.26 -> ~0.05-0.10, crossover measured either way. And fp8 stops being a speed lever: the run measured a bf16 talker instance at 2678 MiB (free went 3605 -> 903, minus the 24 MiB calibration transient), while a 3080 has 2551 MiB usable above the corridor. bf16 misses by 127 MiB on both 3080s, so fp8 is the GATE on the talker pool existing at all, not an optimisation after it. DESIGN_488_talker_pool_and_revision_guard.md carries the rest: TRT engine discipline (I/O signature, dynamic-shape profiles, sha256 + consumer matrix, separate sm120/sm86 engines), the pool with in-order assembly and the head-biased dispatcher a straggler makes necessary, the always-eager chunk policy with the lag-driven context-adaptive clause rule written as an explicit rule rather than hoped for as a side effect, the two-stage revision guard (deterministic prefix check first, Qwen only on divergence, unparseable verdict treated as no-break because a false repair is worse than a missed one), the RAM-cached correction prefix keyed (voice, language) and invalidated on roster events, and the four server interfaces agent 8's client halves attach to -- of which only the per-turn abort channel is genuinely new protocol. Tests: 29 hermetic green (16 instrument + 13 lane). The headroom fixture is now derived from the constants after the calibration transient was resized 96 -> 24 MiB and a hardcoded fixture stopped exercising its branch. ruff + codespell clean.

…rtion says why The 2x PASS was on 55ec9fe802. The ?raw=1 capture switch landed after it, so the served build is ac5163c54a and it was re-gated: 2 PASS / 3 FAIL. Every failure is one assertion, and it is not in this cut: turn 2: 1 of 2 streamed clauses arrived AFTER the first audio frame (19.13: text must not wait for audio); margins [2.2] It fires when synthesis is unusually FAST -- the failing turns recorded tts_first_audio 1.04s against a 5.02-6.62s median in the same run, so the second streamed clause lost a race it normally wins by seconds. tts_first_audio spans 1.04-6.62s for one identical clip across five runs, which is not a property of a client that counts frames. The sgl-project#488 talker agent is running a ~6 min TTS profiling call inside this tenant. This is 17.8.13 repeating verbatim: gating in that state measures the neighbour. The delta between the green build and the flaking one is two lines (?raw=1 plus one diagnostics field, default behaviour unchanged) and cannot reorder clauses against audio. ac5163c54a is therefore NOT claimed as gated; the live 2x is owed once the tenant is quiet. The hermetic arms are unaffected because they never touch the GPU: mic recovery, hidden gate, roster touch, roster and auto-scroll are green on exactly this file with their sabotage controls still red. Also recorded rather than left to be discovered: client_gate never taps a roster speaker button, so pinnedSpeaker stays null and the re-arm is a no-op for the whole run. The mitigation is covered by reading the code, not by a measurement, and an arm that taps a speaker button and counts speaker.arm frames per turn is owed with it.

…ections that move the plan SLICE 1, BUILT. models/qwen3_tts_fast_predictor.py replaces the single call that the precursor measured at 104.44 ms of a 142.78 ms frame -- 73 % of it -- with a raw 15-step loop. It is a drop-in: the reference reads .sequences off code_predictor.generate (modeling_qwen3_tts.py:1671) and gets the same shape, so the 1596-line prompt builder, the trunk loop and the codec are untouched. install()/uninstall() keep the reference reachable in the same process, which the identity gate needs and an operator needs as a way out that is not a restart. It holds no weights, so it costs no VRAM. Sampled tokens never leave the device -- every .item() there is one of the 602 syncs per frame this removes, and a host round trip would also make the loop uncapturable in slice 2. The step schedule is data, not a comment: prefill takes lm_head[0] and no codebook embedding, decode step g takes codec_embedding[g-1] with lm_head[g]. Off-by-one there does not crash, it picks a neighbouring codebook and degrades timbre, so step_schedule() is pinned by five assertions. The sampler is the main correctness surface and is driven against transformers' OWN TemperatureLogitsWarper / TopKLogitsWarper / TopPLogitsWarper on fixed logits, mask and values separately, including the production defaults (top_k=50, top_p=1.0, temperature=0.9) and all three no-op cases -- a skip transformers takes and we do not is a silent divergence on a DEFAULT config. One test was wrong and is corrected rather than deleted: temperature and top_k genuinely COMMUTE (monotonic scaling cannot change which tokens top-k keeps), so an ordering bug between those two is unobservable; the detectability arm now uses top_p, which does not commute with temperature. Both facts are pinned. 14 hermetic, 215 green across the models suite. End-to-end greedy code-token identity needs the real checkpoint and the tenant's import shims, so it runs with the GPU measurement rather than here. The gap is named in the test file, not only in this message. CORRECTION 1 -- the 127 MiB is a rebooking and fp8 is NOT the pool gate. My "fp8 is the gate" is withdrawn. But two named levers do not reach this boot, read at source: sgl-project#330's dial needs --enable-vram-dial, which is a BOOT flag the deployed launch line does not carry (/proc/3953200/cmdline), so "no restart" does not apply to this instance; and it additionally requires weighted uneven DCP (vram_dial.py:1110) while the boot has no --rank-kv-ratio at all. The knob that DOES reach it is already in the launch line: --rank-auto-reserve-mib 13000,3800,3800, whose two 3080 entries go 3800 -> 4200 for ~400 MiB each at the next restart, with no new flag and no dial precondition. Priced honestly: 16 full-attention layers x 4 kv heads x 256 x 2 x 1 B fp8 = 32 KiB/token, so 400 MiB costs 12,800 tokens on that rank, ~4.9 % of one 262144 context. bf16 instances then fit and the pool does not wait for fp8. Still to check before any quantisation effort: an sm86 bf16 engine may already fit under 2551 MiB by itself, which is a measurement on the built engine, not a prediction. CORRECTION 2 -- TRT is cut as fork lane infrastructure (sgl-project#337), not a talker side path. The discipline I specified already EXISTS in video_enhance/engine_cache.py: EngineKey over nvml_uuid|device|driver|trt| onnx_sha256|precision|shape triplet|flags, provenance manifests, and "no manifest means absent, not usable". So the plan is promote, not duplicate: lift it to srt/trt_lane/ with video_enhance as its first existing consumer and byte-identical video behaviour as the acceptance condition, then add the sgl-project#337 rung registry, sgl-project#286 ledger registration so a loaded engine is parkable like any other VRAM tenant, sgl-project#407 tier placement, per-regime crossover as a reported output, and sm86 refusals by name. The talker's two engines are the first production consumer; RIFE/SR (sgl-project#469) attach afterwards. Ordering unchanged: graphs first -- the 6.53x needs none of this.

…t-order finding that is not where it was reported The standalone profile path puts a ~2.6 GiB second copy of the talker on a card a live tenant and a serving rank already use, and on 2026-08-04 that cost minutes of VRAM triage on an unattributed pid -- this run. It now renames itself sglang::488-talker-profile-GUEST in both /proc/<pid>/comm and /proc/<pid>/cmdline, so ps, top, py-spy and nvidia-smi attribute it on sight. GUEST is deliberate: it answers the second triage question, which is not whose process this is but whether it belongs on that card. Best-effort by two mechanisms (setproctitle for argv, prctl for comm) because a diagnostic aid must never kill the run it is diagnosing. Pinned as a BINDS-PROOF rather than "the mechanism exists" (sgl-project#493): the test calls the tagger and reads the result back out of /proc, and separately asserts the ticket number survives the kernel's 15-byte comm truncation -- outside that window `top` would show nothing useful. 20 hermetic, green. The in-process path needs no marker; it runs inside the tenant's own pid.

…a gate that had to be rewritten SLICE 2, BUILT AND MEASURED. models/qwen3_tts_graph_driver.py captures the talker's token loop: fifteen predictor step graphs over a 16-slot static scratch (one per residual group, because a graph cannot switch which lm_head runs) and the 28-layer trunk decode step as one graph. Frame time 142.18 -> 14.64 ms, RTF 1.706 -> 0.176, 9.71x; the 84.7 % launch gap the precursor measured is now 4.6 % on the trunk and 5.9 % on the predictor. Artefacts: /spinning/gpu-battery-results/2026-08-04_488_slice2_graphs/. Two independent runs; the graphed rung was 14.64 ms both times while the eager reference moved +/-4 %, which is what an overhead-bound path inheriting host jitter looks like. No transformers patching was needed, which is worth stating because the alternative was reimplementing the layer stack. _preprocess_mask_arguments:840 returns an already-4-D attention_mask AS-IS, so a caller-owned static mask buffer reaches sdpa untouched through both the trunk's single-mask path and the predictor's dict path; passing cache_position and position_ids explicitly removes the two host-side aranges; and the one real sync on the decode path (cache_position[0] == 0, modeling_qwen3_tts.py:1696) lives in the OUTER forward, which we never capture. StaticLayer.update already keeps its write pointer as a device tensor advanced in place, so a replayed graph walks forward through the cache by itself -- refuse_unless_graph_safe pins that instead of trusting the version, since a Python-int spelling would silently rewrite one slot forever. The two graphs need OPPOSITE discipline and getting it backwards is silent. A predictor step owns one slot for the life of the process, so its write-slot pin is captured INSIDE the graph, which makes each step idempotent and deletes the between-frame rewind. The trunk's slot advances every frame, so its pin must sit OUTSIDE the capture -- captured, it would freeze the trunk on one slot for the whole utterance. Found by reading before running, not by debugging audio. GATE 1 AS SPECIFIED IS NOT ACHIEVABLE, and the cut is still correct. ANALYSE_488 7.5 asked for identical codec tokens against the eager reference; measured, 23.3 % differ. Decomposed, the design had conflated two changes: eager-static vs GRAPHED is bit-exact (0.000000), and the whole divergence comes from the StaticCache + explicit mask that capture requires. Rerun in float32 that same pair agrees to 1.4e-6..2.4e-6 relative against fp32 eps 1.2e-7 -- reduction order, not semantics: an explicit mask makes sdpa pick a different backend, and at bf16 a few ulp flips the predictor's frequently near-tied argmax. So the gate is split -- capture gated bit-exactly against the same static cache (stronger than the original for what capture can break), semantics gated in fp32 where the question is answerable, and the 23.3 % reported up front so the listening arm is told rather than surprised. check_static_cache_semantics.py is that falsifier, kept executable rather than quoted. Slice 1 is corrected on the record too: the raw loop alone bought 1.04x, not the estimated ~1.3 RTF. The 602 syncs were not the cost. Its value was enabling capture. CORRIDOR: the FIRST run peaked ~5090 MiB against a 3000 MiB estimate and left the 5090 at 312 MiB free, under the 400 MiB floor -- transient, no serving process affected, and it should not have happened. The line items were right and the total was wrong: each phase's caches, graph pools and cuBLAS workspaces were still resident when the next allocated. Phases now release before the next claims, and the corridor is re-checked BETWEEN phases and aborts, not only at the ends where a mid-run peak is invisible. Re-proven: minimum 1382 MiB free, all three cards ok at every phase. Also recorded, contradicting a claim carried into this session as fact: the reserve rebooking to 4200 shipped, but the 3080s' free VRAM is 2957/2959 MiB -- 6-8 MiB above baseline, not 400. Usable 2557 against a 2678 MiB bf16 talker, so the pool gate is NOT open and item (3) is blocked on 121 MiB. 19 hermetic tests for the geometry, the mask, and four refusals that each corrupt audio without raising (int write position, sliding-window layer, shared-pool replay out of capture order, replay before capture) -- each exercised until it fires. 66 green across the 488 suite; ruff and codespell clean.

…, the baseline did not MECHANISM, traced to the code that actually ran. First correction: that code is /spinning/wt-530-serving/python, reached via PYTHONPATH in /proc/1236/environ -- NOT the htsglang-gpu tree the venv's editable install points at, which is a detached HEAD from 2026-07-13 predating --rank-tp-ratio auto-performance and unable to parse this cmdline. Reading the wrong tree would have produced a confident wrong answer. The reserve entered the arithmetic exactly as designed; the boot logged "derived memory budgets [19607, 16280, 16280] MiB ... (reserve per GPU: {0: 13000, 1: 4200, 2: 4200})". budget = NVML total - reserve, converted once to a per-rank fraction and consumed as the KV-pool ceiling. What the knob does NOT do is cap allocation, and every rank overshoots it: +686 and +684 MiB on the two 3080s, +2963 MiB on the 5090 (CUDA context, NCCL buffers, flashinfer workspace, graph capture, prefill scratch). That was already on record here before the prediction was written -- server_args.py:10758-10763 "It does NOT cap any runtime allocation", and NOTE_493 section 3 measuring the same class on another model. But the overshoot is a CONSTANT, not a function of the reserve, so free VRAM moves 1:1 with it. Reconstructed with the same 686 MiB: reserve 3800 -> free 2557, reserve 4200 -> free 2957, and 2957 is what the card shows. The 400 MiB DID materialise. The false premise was the BASELINE: the "2951 MiB free at 3800" the prediction reasoned from sits within 6 MiB of the post-change reading, which is what a number taken on a boot that already carried 4200 looks like. Stated with its limit: the pre-reboot log did not survive the container restart, so "the old boot's overshoot happened to be 400 MiB larger" cannot be excluded -- and does not matter, because both readings give the same present state and the same lever. The lever is priced rather than argued: free and KV budget move 1:1 against each other, so 4400 buys a 79 MiB margin for ~6,400 tokens and 4600 buys 279 MiB for ~12,800, against a 333254-token pool -- if the 3080s are the min-reducing ranks, which is NOT established and is flagged as such. CONSEQUENCE, and it is the part that should change the plan: slice 2 devalued this pool by ~10x. Its justification was head-of-line blocking of a median 4.8 s synthesis (inprocess_tts.py:124-134); at RTF 0.176 that median is ~0.5 s and one instance sustains ~5.7x real-time. The pool's remaining value is multi-user concurrency, not single-user latency, and the live case driving this work is one user. Recommendation recorded: do not spend the KV tokens, leave the reserve at 4200, leave the pool unbuilt, revisit if concurrent speakers become real. The 121 MiB is not a blocker to route around -- it is a price signal on something that stopped being worth buying. Docs only; 66 green, codespell clean.

…arger than estimated STEP A BUILT AND SMOKE-GREEN. GraphedPredictorFrame.install() swaps code_predictor.generate for fifteen captured graphs and nothing else changes -- worth 2.98x on its own (frame 142.18 -> 47.64 ms, RTF 1.706 -> 0.572, a 5511 ms clause becomes ~1850 ms), with the remaining 3.3x waiting on Step B. Handoff is documented in DESIGN_488 4b: sampling arguments, refusal/fallback table, the measured VRAM cost, and gate_loaded_model() to run in-process against the live tenant. The tenant is NOT touched by this commit. A replayed graph cannot draw randomness, so entropy became an input: UniformPool keeps pre-drawn uniforms on the device and advances its cursor INSIDE the last step's graph, which costs the host nothing per frame. Explicit uniforms are still accepted, written under the cursor, so one captured body serves both production and the identity gate. The sampling arguments are required keyword arguments with no defaults, because a graph bakes its warpers -- and because of a trap this deployment invites: the tenant passes temperature=0.9, top_p=0.9, and those are the TRUNK's knobs. The code predictor runs on subtalker_* (modeling_qwen3_tts.py:2036-2039), which the tenant never passes, so top_k=50, top_p=1.0, temperature=0.9. Capturing the tenant's 0.9 would apply a warper the reference never applies -- permanent, silent, audible only as timbre. install() reads the reference's own signature and refuses at load time; the refusal has a can-fail test. A runtime mismatch FALLS BACK to the reference instead of raising, a deliberate exception to fail-fast: the reference output is correct, just slower, so raising would break a live turn to prevent what the fallback already prevents. It is counted, so telemetry can say the speedup is not being delivered. GRAPH OVERHEAD MEASURED, and ANALYSE_488 7.4 understated it by 4.9x: 1184 MiB, not 242 -- predictor 183.5 against ~60, trunk 1000.5 against ~181. Identical on sm86 and sm120, so it is allocator- and shape-driven, not architectural. The discriminator is complete: the trunk term is LINEAR in cache slots (1024/256/64 -> 1000.5/262.3/77.8 MiB), fitting 0.96 MiB per slot + 16 MiB against a static cache of 0.109 MiB per slot. Capture retains every per-layer expanded-KV intermediate rather than freeing it, so the cut costs ~8.8x the cache it captures. Slot count is thus the dominant VRAM knob of the whole feature, and 1024 is the wrong default for a ~3.2 s clause; right-sizing to 256 is worth 738 MiB, six times the pool's entire original 121 MiB shortfall. The default stays 1024 only until generate_icl_prompt's real prompt length is measured. POOL VERDICT RETRACTED, kept in place rather than deleted because the way it was wrong is the lesson. It priced the pool on single-synthesis latency, found graphs had taken that win, and called the pool devalued ~10x. The axis it never checked: a turn is not one synthesis -- session.py:1852-1871 already splits it into clauses, so the pool parallelises WITHIN a turn and composes multiplicatively with graphs. Rule recorded: a building block is devalued only after checking every axis it serves, not the first one. The honest re-pricing (2.2f/2.2g) is worse for the pool than the retracted verdict was: a fully graphed instance misses a 3080 by 567 MiB at 256 slots, not 121, because the graphs cut is itself what put it out of reach. Three unblock paths priced; recommendation is to shrink the instance (quantise + right-sized cache), not to tax serving KV. Dispatcher gains a HARD acceptance gate (2.3): turn-scoped cancel across running, queued and already-delivered clauses, with four observable conditions, plus the arithmetic for why drop-and-refill beats salvage at RTF 0.176. 80 green, ruff and codespell clean.

merrymercy added 2 commits May 29, 2024 18:31

do not set the default value of global random seed so the server can …

6121553

…be random by default

update

f7413f3

merrymercy merged commit 80a33ce into main May 29, 2024

merrymercy deleted the fix-random branch May 29, 2024 22:41

merrymercy changed the title ~~Do not set the default value of global random seed~~ Make the server random by default May 29, 2024

Ying1123 added a commit that referenced this pull request May 31, 2024

Revert "Do not set the default value of global random seed (#488)"

80d365e

This reverts commit 80a33ce.

Ying1123 mentioned this pull request May 31, 2024

Revert "Make the server random by default" #492

Merged

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Do not set the default value of global random seed (sgl-project#488)

7c1d503

EdwardXuy pushed a commit to shun8686/sglang that referenced this pull request May 9, 2026

Merge pull request sgl-project#488 from hhhh1252023/patch-1

d784fec

Update full-test-npu.yml

efschu added a commit to efschu/htsglang that referenced this pull request Aug 4, 2026

sgl-project#488 codespell: unparseable -> unparsable in the guard spec

f56abf6

efschu added a commit to efschu/htsglang that referenced this pull request Aug 4, 2026

Merge feat/talker-lane-488: native talker lane (sgl-project#488)

e08946d

Navigation Menu

Repository navigation

Make the server random by default - #488

Make the server random by default#488
merrymercy merged 2 commits into
mainfrom
fix-random

merrymercy commented May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

merrymercy commented May 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant