docs: fix typo - #742
Merged
Merged
docs: fix typo#742
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cen121212
pushed a commit
to cen121212/sglang
that referenced
this pull request
Nov 10, 2025
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.ai to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. --> ## Benchmarking and Profiling <!-- If this pull request impacts inference speed, provide benchmarking and profiling results. --> ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
cherryblo
added a commit
to cherryblo/sglang-project
that referenced
this pull request
Jul 2, 2026
Fixup test_npu_large_max_new_tokens.py,test_npu_lora_overlap_loading.py
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…t is a build Ordered step (1) says read the refusal's reason before touching it. Read, with blame: the answer is a third option, neither of the two the brief anticipated. NOT a sgl-project#718-class safety guard. git blame puts it in 05933d0 (2026-07-13), the flag's OWN introducing commit, listed among ordinary validations ("hierarchical/unified radix tree rejected") with no measured failure behind it. NOT merely never-composed either. The unified mamba component contains no reference to mamba_checkpoint_interval, is_on_interval or mamba_track_interval, while MambaRadixCache consults the grid in six places. So composing the flags today would corrupt nothing and would leave --mamba-checkpoint-interval SILENTLY IGNORED while its help promises deterministic absolute-multiple positions. That is the sgl-project#742 class exactly -- and lifting the guard would replace an honest refusal with a dishonest acceptance, which is worse than leaving it. The stated reason is also aimed at the wrong class: it blames HiMambaRadixCache, which per NOTE_745 has no construction site (registry.py:107-112 routes hybrid-SSM + hicache to the unified tree). Second time that aiming error has surfaced in this area. Scoped instead of forced: the lift is a mirror of the grid at five seams of the unified component (match validator/finalize, commit_insert, prepare_for_caching, split redistribute, evict/drive_eviction), each named with the mamba_radix_cache.py line it mirrors. The eviction seam is the subtle one and it gets EASIER under the unified tree. With the interval set, MambaRadixCache spares the deepest anchors per path because "losing the deepest one silently moves the resume point of identical requests and re-introduces run-to-run drift" (mamba_radix_cache.py:1092-1097) -- a determinism contract that exists because a spilled anchor is a DEAD anchor on a device-only pool. With a host tier an evicted anchor is still a valid match and loads back (mamba_component.py:71-74, :139-144), so the protection window can be weaker here, not stronger. That is exactly the directive's goal: 8k deterministic anchors that survive on disk. Recommended shape recorded for when it is built: cadence 8192 (16 x chunked_prefill_size), anchors host-tier-eligible with no separate pin class, red-first with a hermetic construction smoke plus an anchor-reaches-host-tier test, and the refusal deleted only in the same commit that makes the grid effective. Method precedent: sgl-project#547 -> sgl-project#550, reading a describing-not-reasoning refusal against the tree to tell impossibility from unbuilt. Docs only; nothing boot-verified. Post-boot branch, not for the closing train.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
… grid is now real on the unified lineage --mamba-checkpoint-interval composes with --enable-hierarchical-cache (and with SGLANG_ENABLE_UNIFIED_RADIX_TREE): both server_args refusals guarded the same unified mamba component, which now mirrors the checkpoint grid at every decision MambaRadixCache makes -- match gating, retention, cache_len, branching, eviction -- through shared rules in mamba_ckpt_utils.py (seam map + per-seam status in docs/dev/NOTE_747_mamba_ckpt_hicache_refusal.md par.7). Lifted only now that the build exists: deleting the refusal earlier would have shipped a silently-inert flag (the sgl-project#742 defect class; NOTE_747 par.3). What the composition buys (the point of sgl-project#747): interval anchors are host-tier-eligible like any other retained node, so a deterministic anchor survives device eviction -- it stays a valid match and loads back -- and can reach disk through the storage backend. Cadence for 8k anchors: 8192, which requires --chunked-prefill-size >= 8192 (the interval must not exceed the chunk budget; NOTE_747 par.5 first draft had this constraint backwards and is corrected). Unchanged and deliberate: - enable_int8_mamba_checkpoint x hierarchical stays refused (separate guard, int8 states have no host-pool layout). - All grid validations (chunk multiple, page multiple, chunked-prefill ceiling, track-interval unification) run identically with hierarchical cache on -- pinned by the new composition tests. test_hierarchical_cache_rejected flips into two pins: the combination is accepted (and still unifies the tracking grid), and the grid rules still reject an off-chunk interval with hierarchical on. Boot-gated acceptances are named in NOTE_747 par.8 (anchor observed surviving to host/disk and resuming, serving boot with the composed flags) and ride a later WINDOW_TICKET_745 arm; ARM-1 boots hierarchical without the interval by design. Tests: mem_cache suite + truncation-align + hicache-hybrid-gdn + mamba-evict-floor: 1239 passed / 1651 skipped / 183 subtests, hermetic (CUDA_VISIBLE_DEVICES empty, worktree PYTHONPATH). ruff/codespell: no new findings (server_args.py 358 pre-existing before and after).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
ROOT, corrected against the handover's framing. The handover read the store
census as "15457 KV pages vs 11 mamba anchors = grid behaviour" and ordered
the anchors published at every write-through. Measured on the same boot
(boot_855_1028fence), that premise is a PAGE-vs-NODE denominator mix and the
ordered fix is already in place:
* `#969H BACKUP` = 33 lines = n=1..11 on EACH of PP0/PP1/PP2 at identical
timestamps. The probe logs every call up to n<=40, so 11 lines means
`write_backup` reached its component loop exactly 11 times per rank in the
whole run -- and all 11 carried `mamba_value=has_value`.
=> mamba coverage OF THE HOST-BACKUP PATH is 11/11 = 100%, not 11/15457.
* All 11 `.mamba` hashes in the store are also full-KV page hashes
(intersection 11, mamba-only 0): one anchor per node at a real page
boundary, shared key namespace.
* The 15457 KV pages are the PAGES of those same 11 nodes:
`write_backup_storage` writes `keys=node.hash_value` (every page of the
node) while the mamba branch writes `keys=[node.hash_value[-1]]` (one
trailing page).
The real hole is one link earlier and is not mamba-specific: `_inc_hit_count`
returns before any backup when `chunked=True` (upstream's "skip the hit count
update for chunked requests"). Under chunked prefill NOTHING is published per
chunk -- so the 11 backups are the 11 FINISHED requests, and a chunked prefill
that never finishes publishes nothing at all.
That is why the 13179-token prompt found its deepest anchor at 3072 and
recomputed the remaining 10107 tokens: the anchors are as dense as finished
requests, not as dense as chunks.
FIX: allow a chunked-prefill node to reach the host tier, gated structurally
on (storage tier present AND a MAMBA component present). The per-chunk node
already carries a donated state on the device
(`MambaComponent.prepare_for_caching_req`, is_finished=False branch); this
early return was the only reason it never reached host or storage.
UPSTREAM-MINIMAL: the chunked skip IS upstream (`hiradix_cache.py`), so this
is a DEVIATION and carries its burden of proof. Named: (1) drain-and-flip --
a chunked prefill interrupted by a flip never reaches `cache_finished_req`
(sgl-project#856 removed the carry, the flip DISCARDS it), so upstream's publish-at-finish
never fires; (2) GDN hybrid -- a recurrent state is valid at exactly one token
position, so the per-chunk anchor has no pure-attention analogue. Gate off =
byte-identical to upstream (verified: branch truth table, 0 mismatches over
all 8 chunked/write_back/force combinations).
LAWS: `mamba-per-knoten-nicht-gitter` in its own words ("states per radix
node/chunk like KV pages"), and it waives write volume explicitly.
`kein-doppel-prefill` (sgl-project#939): loss bound becomes ONE chunk.
sgl-project#968 PP0 DEBT -- WHERE THIS PATH STAYS RANK-LOCAL: the publish decision is
taken at the scheduler's chunk boundary, which every rank runs for the same
request at the same split, so it is unanimous BY CONSTRUCTION, not by
agreement; this path holds no reduce (the `check_prefetch_progress` MIN is
TP-scoped and the boot runs tp_size=1/pp_size=3, so it is structurally
skipped). `raenge-nie-uneins` is met by construction and `#1028P
CHUNK-PUBLISH` is how the claim gets CHECKED: identical n at identical
timestamps across ranks, the evidence shape `#969H` gave for the 11.
The one rank-local input reachable from here is the sgl-project#581/sgl-project#773 write-through
pin budget, which fired ZERO times in that boot (trap-safe: bare 0, genuine 0)
because 11 backups never approached it. Per-chunk publishing makes it
reachable for the first time; `pin_skipped` rides on the same line so a
nonzero count next to a divergent n is the divergence, named in advance.
COST, QUANTIFIED NOT BUILT: one `.mamba` blob is 78446592 B = 74.8 MiB
(measured, all 11 identical); a KV page is 32768 B. A published 4096-token
chunk therefore adds 74.8 MiB of anchor on top of 128 MiB of KV (+58% L3).
Host RAM is UNCHANGED -- the host mamba pool is pre-sized at boot from
`hicache_ratio` (1.5) x device slots, so anchors roll through a fixed tier and
land on disk. The int8-anchor idea (sgl-project#1013) would cut the 74.8 MiB and is
deliberately NOT built here.
ALSO: #1028B FETCH CAP instrument at the `min` in `hicache_storage.py`, the
cap that decides how much of an existing KV prefix a prefetch may claim. It
printed nothing: `final_pages`, `kv_pages`, `boundary=`, `hit_pages` each
occur 0 times in the whole 5.87 MB log, so "anchors too sparse" and "KV prefix
too short" produced the same number and were NOT separable from that boot.
Now both terms print on one line.
Desk checks (error-class matched): ruff F821/F811/F841/E9 clean on both
changed files (new names + new attribute reads); branch truth table executed,
gate-off equivalence exact. The gate is deliberately NOT memoised --
`enable_storage` is False at __init__ and only set in `init_hicache`, and a
memoised early False would leave a wired-but-inert write path, the sgl-project#742/sgl-project#745
class this area has produced before.
NOT YET BOOT-PROVEN. Acceptance is the next boot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.