feat: add chat template for internlm2-chat by zhyncs · Pull Request #802 · sgl-project/sglang

zhyncs · 2024-07-29T15:40:25Z

Motivation

# before
python3 -m sglang.launch_server --model internlm/internlm2-chat-7b --trust-remote-code

# after
python3 -m sglang.launch_server --model internlm/internlm2-chat-7b --trust-remote-code --chat-template internlm2-chat

# client
  curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "internlm/internlm2-chat-7b",
    "messages": [{"role": "user", "content": "介绍广州"}],
    "stream": false
  }' \
  --no-buffer

cc @merrymercy @Ying1123 @hnyls2002
Before this fix, users needed to explicitly pass in stop_str when making a request. Otherwise, the response would not stop.

Currently, users need to specify the --chat-template when starting the server, which is an improvement over before but still not good enough. We should automatically detect the chat or instruct models and determine whether there is a chat template in the tokenizer json, then automatically match and set it.

I plan to implement this feature in another PR, but the priority is not very high at the moment. Assign it to me first, and I will work on it later.

Modification

as titled

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

…ole-flip second count The REFILL instrument printed each leg's duration beside "the ~3.1 s pinned baseline". On 2026-08-22 a briefing built on that line concluded the flip economy was broken, put the cost at 64% of wall clock against an 8x regression, and went looking for a silent host-RAM fallback to blame. The line invites every step of that reading, and the comparison it offers is invalid three ways: SCOPE ~3.1 s is a WHOLE FLIP (NOTE_677_floor_components.md:135-143 uses it as "Against a ~3.1 s flip"), not a refill leg. PATH sgl-project#690 measured the PINNED image path (NOTE_690_gdn_state_spread.md:58-85), which predates the file-backed arm entirely, so it is not a baseline this path ever held. BYTES sgl-project#690 moved 9614.9 MiB/rank; these legs move 8574-16363 MiB, and elapsed time tracks bytes moved (r ~ 0.80 over 45 logged legs), so seconds are not comparable across them. There is no regression and no fallback here. The file-backed arm is an explicit opt-in (--phase-flip-image-file-backed, server_args.py:5818), and weights_arena.py:441-445 states it REFUSES rather than falls back, naming the silently-inert-flag class as the reason. The boot log carries "FILE-BACKED (reclaimable)" nine times and zero fallback warnings. Its help text names what the slower path buys: without it the images are ~68.7 GiB of unreclaimable host RAM on a swapless box and the boot is OOM-killed during init. So the line now reports a rate against a rate and carries the reference's conditions, which are what make a baseline transferable at all. It still says the arm is slower, because it is; what it no longer does is let that read as a defect to hunt. The instrument is extracted as refill_report() so it can be tested without a flip. Behaviour is otherwise unchanged: same call site, same try/except, an instrument may never break a flip. Tests: test_refill_baseline_honesty_802.py, 6 tests, red first (all six failed on the missing entry point, then on the bare second-count and the absent conditions). 55 pass across the refill and sgl-project#796 families.

…the discriminator existed WHAT KILLED THE INSTANCE. 2026-08-22 17:22:44, under load, during a pp_to_tp flip, PP0 raised and took the whole instance with it: KvReshardError: PHASE-FLIP-GDN payload checksum mismatch from peer 1 (stored -4664535355886616603, computed 15255426930); refusing to scatter `stored` is NEGATIVE. `uint8_checksum` is an exact int64 sum of UNSIGNED bytes, so its value lies in [0, 255 * nbytes] and nowhere else. A negative value is not a checksum that disagrees -- it is not a checksum at all. The trailer was never written. `computed`, by contrast, is entirely ordinary: 15255426930 requires a payload of at least 57 MiB, which a GDN leg is. So no comparison of contents ever took place, and the instance was killed for a data corruption that had not happened. The log line then sent every reader to look for one. THIS IS THE SECOND TIME, AND THE FIX WAS ALREADY WRITTEN. Sitting directly beside `uint8_checksum` in weights_arena.py is `checksum_is_representable(value, nbytes)`, whose docstring records sgl-project#656 register C22 as exactly this misreport -- a negative "sender" checksum of -4450328002521349435 -- and states the distinction in full: BOTH values in range -> the two ends framed the payload the same way and computed different sums over it. The DATA differs. A value OUT of range -> that field was never a checksum. `GdnFlipMover._verify` compared first and never asked. Mechanism present, actuator missing -- the same shape as sgl-project#800's stash with no escape and sgl-project#800's probe that answered None for two different states. WHY THE LENGTH CHECK CANNOT SEE IT. `kv_reshard._dist_exchange` allocates the receive buffer with `torch.empty` and never pre-zeroes it, and the send/recv byte counts are derived independently on each rank from replicated geometry and never handshaked. An under-filled receive therefore keeps its allocated `numel` -- so `_verify`'s length check passes -- while the tail, which is exactly where the trailer is read from, still holds the original allocator garbage. `bounded_collective` polls only `is_completed()`; it never checks that the bytes actually landed. WHAT THIS CHANGES, AND WHAT IT DOES NOT. The two failures now have two errors: an out-of-range trailer is reported as a MISSING TRAILER, naming the transport and the framing and saying in as many words that this is NOT evidence of data corruption; an in-range but different trailer is still reported as a checksum mismatch, now stating that both values are representable so the DATA is what differs. This does NOT stop a receive from being under-filled -- the physical root (a GB-scale receive posted while PP0 sat at 1021 MiB free, one second after spill rung 1 returned 368/128/128 MiB across the three ranks, with cuMemCreate refusing against the 1024 MiB corridor floor) is still open. It makes the next occurrence say what actually happened instead of the one thing it could not have been. WHY THIS NEVER FIRED BEFORE. Without load the GDN mover has nothing to carry: the 17:21 flips logged "PHASE-FLIP-GDN moved 0 slot(s): sent 0.00 MiB, received 0.00 MiB". The crash flip was the first with real state in it, which is why six clean flips preceded it. TESTS (CPU only, no GPU, no process group): test_gdn_payload_trailer_802.py 7 passed (new) test_gdn_flip_tree_slots_767 + kv_reshard_headroom_363 + kv_reshard_stuck_arm_363 + seam_arena_tail_additive_656 57 passed total CAN-FAIL, both directions, one broken build at a time: * representability test removed -> the two missing-trailer cases go red (the misdiagnosis returns), the real-mismatch case stays green; * representability test always firing -> the control, the real-mismatch case and the boundary case (0 and 255*nbytes, which ARE representable) go red. Both mutants killed, file restored and verified after each.

… the refusal too THE PHYSICAL ROOT of the unwritten trailer, and it is not memory pressure. `_dist_exchange`'s `incoming_nbytes` is a PREDICTION. The caller derives it locally -- `gdn_flip_mover.move` computes `_pair_nbytes(..., len(slots), ...)` from its OWN rank-local slot enumeration -- while the sender packs its payload from ITS OWN. For the GDN leg that enumeration is `flip_mamba_slots`, "resident requests' mamba slots UNION the radix tree's checkpoints", which has NO cross-rank agreement step at all. Its own docstring contrasts it with the KV leg's `build_flip_live_slots_fn`, which does agree by union; sgl-project#767 fixed what this leg enumerates but left it rank-local. On 2026-08-22 the three ranks had enumerated 150695 / 159848 / 151656 rows, and PP1 -- the peer whose payload failed -- had spent the preceding six seconds in a void livelock (2353 x #797d paired with 2353 x sgl-project#798, on three real retractions; the healthy boot ran 5 voids on 5 retractions). FOUR INDEPENDENT MISSES let a short receive through: * the receive buffer is `torch.empty` and is never zeroed; * NCCL p2p with mismatched counts does not raise; * the receiver's length check passes -- an under-filled buffer keeps its ALLOCATED numel; * `bounded_collective` polls `is_completed()`, never a byte count. So the tail keeps allocator garbage exactly where the checksum trailer is read, and the flip died reporting a corruption that had not happened. THE FIX is a size agreement before any byte moves: one zero-filled matrix summed across the group carries both halves -- what each rank will SEND to each peer and what each rank has SIZED ITS RECEIVE for -- and every rank checks every ordered pair against the identical matrix. It runs unconditionally and BEFORE the `if not ops` early return, because a rank that skipped it while its peers entered it would be the desynchronisation this exists to prevent. THE REFUSAL IS COLLECTIVE, and the first cut of this guard was not. Its own three-process test caught that within a minute: checking only "what MY peers advertised versus what I expected" makes the verdict rank-local, so in a three-rank group where rank 1 packs short, ranks 0 and 2 raise while rank 1 sees nothing wrong, walks into `batch_isend_irecv` and blocks for ever on peers that have already left. A guard that strands the group is worse than the corruption it prevents. Both halves are now gathered and every rank reaches the identical verdict off one shared matrix. RESIDUAL RISK, STATED PLAINLY. `_dist_exchange` serves three callers -- the GDN mover, the flip seam leg (phase_flip_runtime) and the KV reshard leg -- and the guard now applies to all of them. That is intended: a p2p pair whose two ends disagree on the byte count is never legitimate. But it cannot be exercised on metal from here. If "SIZE DISAGREEMENT" fires on the KV or seam leg at the next boot, that is a second, previously invisible instance of this same defect and not a regression of this change -- the previous behaviour was to receive short and read garbage. TESTS (CPU, real three-process gloo group, driving the SHIPPED `_dist_exchange`): test_reshard_size_agreement_802.py 2 passed (new) test_gdn_payload_trailer_802 + gdn_flip_tree_slots_767 + kv_reshard_headroom_363 + kv_reshard_stuck_arm_363 + seam_arena_tail_additive_656 57 passed CAN-FAIL: the mutant that restores the rank-local verdict (`receiver != me`) does not fail the suite -- it HANGS it, which is precisely the defect. Bounded demonstration: the mutant does not complete within 60 s; the fixed build finishes the same case in 5.2 s. The empty-exchange case is the second direction: it pins that the collective still runs when a rank has nothing to move, which is the deadlock this change could otherwise introduce.

…n a send nobody owes The PP output ring wedged with all three ranks alive and none of them able to move (specimen /spinning/evidence-665-f1/wedge_802f_1712/, PP=3, --enable-phase-flip, py-spy of all three schedulers): PP0 _pp_commit_comm_work <- _pp_commit_pending_req_work (:4071/:2262) PP1 _pp_recv_dict_from_prev_stage <- _do_recv (:5608/:6069) PP2 PpChainReceiver.recv <- recv_requests (pp_chain_receiver.py:329) PP0 is flushing the request chain, which PP1 can only take at the top of its next pass; PP1 never reaches that top because it is blocked in the output receive; PP2 waits for the chain send PP1 has not reached. Three arcs, one cycle, no timeout anywhere. THE ASYMMETRY. The two ends of the intermediate hop apply unrelated predicates. `_do_recv` decides to receive from THIS rank's own slot state; the non-last sender in `_pp_send_output_to_next_stage` decides to forward on `if pp_outputs:`, which is whatever it received LAST iteration. The last-rank hop is matched by construction because it consults `_pp_output_expected_for_slot` -- but that flag is the FIRST rank's verdict, published for PP0's arc. Nothing publishes the same thing for the intermediate hop. WHY THE EXISTING VOID CONTRACT DOES NOT COVER IT. `pp_void_forward_ payload` (sgl-project#797) already forwards a void along this hop and stops at `pp_first_retracting_rank`, on the argument that rank r and everything after it has an empty slot whose receive early-returns. That holds for the VOIDED slot -- `_pp_void_own_batch` empties it. It does not hold for a slot the resident decode path still occupies, and both void paths keep resident requests on purpose (`_pp_absorb_void_output` refuses to release them as a double-free; `_pp_void_own_batch` deliberately leaves `running_mbs` alone). The specimen shows exactly that: PP0's last act was absorbing a void for slot 0, leaving `pp_outputs` None, while PP1 logged `running=1 chunked=1` and re-entered the receive for a slot PP0 would never send to. PP0 cannot know PP1's resident set, so closing this needs a per-slot expectation every non-first rank publishes to its predecessor -- a protocol extension, and not this commit. WHAT THIS COMMIT DOES. The sgl-project#789 readiness gate is parameterised by wire kind and the output receive now passes through it. The CHAN_DICT counter was never proxy-specific -- one counter per wire, demultiplexed by `__msg_type__` after it comes off -- so the gate reads the same true statement about the same wire either way; `kind` selects only the per-(src, kind) inbox peek. The ring is cut at its one cuttable arc: PP1 waits boundedly and refuses by name instead of for ever. It does not make the missing send appear, and the error text says so. AN ALIAS, NOT A WRAPPER. `_pp_wait_for_proxy_readiness` is now a class-level alias for the same function object. About ten stand-in holders across the sgl-project#631/sgl-project#757/sgl-project#787/sgl-project#789/sgl-project#791/sgl-project#795/sgl-project#797/sgl-project#798 test family bind that name one method at a time; a delegating wrapper resolved the second name on the HOLDER and turned 9 green tests into 5 failures. Measured, then fixed. Tests: test/registered/unit/managers/test_pp_output_readiness_ring_802.py -- three real gloo processes, real PhaseFlipCounters, neutering done in the child (spawn re-executes the module, not the test body). * red arm, gate neutered: stuck_ranks == [0, 1, 2], specimen reproduced * green arm: stuck == [], PP1 reports "sgl-project#789 OUTPUT READINESS TIMEOUT" and "sgl-project#802-ring", PP0 chain-flushed, PP2 requests-received * false-positive direction: upstream really posts -> receive succeeds * no-op without counters, so the non-phase-flip default path is unchanged Mutants, both die: call site removed -> green arm red (all ranks stuck); gate raises unconditionally -> false-positive test red. Regression: baseline 787+789 = 9 passed / 0 failed; with this change 791b + 789 + 787 + 802 = 17 passed / 0 failed.

The fix (2bcb868) is unchanged and still right. Two things it SAID were wrong, and both would have cost the next reader real time. 1. "AN NCCL GROUP" -- no. On this rig every process group is barlink-bar1: world:0, pp:0, flip_tp:0 and flip_dcp:0 each log "requested=bar1, ACHIEVED=bar1", and PyNccl communicator construction is skipped outright (9 times in the 18:58 boot). "No backend type associated with device type cpu" is the GENERIC torch.distributed error for a device-only group handed a CPU tensor; it reads identically under bar1 and under NCCL. The defect was tensor placement and never the transport choice. Calling it an NCCL group invites the next reader to hunt a barlink-standard violation that did not happen. (The one "falling back to NCCL" line in the log is the custom-allreduce layer only, superseded by the barlink takeover, and tp_size=1 on this boot.) 2. "EVERY TEST INJECTED all_reduce, SO THE DEFAULT PATH NEVER RAN" -- also no, and this one was my own guess dressed as a finding. Checked instead of assumed: `grep -c 'all_reduce=' test_mamba_slot_union_801.py` is 0, and all five cases call agree_mamba_slots(local, dist.group.WORLD, ...) over a real process group. The boot path WAS exercised. The real gap is better than the guess. That group is init_process_group("gloo"), and gloo ACCEPTS CPU tensors where the production device-only group refuses them. The collectives really ran -- on a backend that could not refuse the thing that later refused them. So "exercise the default path" would not have caught this: it was already being exercised. The generalisation worth keeping: a hermetic harness must be, in every dimension it claims to cover, at least as strict as the production path. A more permissive stand-in does not weaken a test, it inverts it -- green becomes evidence that the strict case was never posed. The recording group in the new test is therefore deliberately stricter than gloo, which is precisely the strictness the gloo harness lacked. Also documents, without changing, that the two header reads are host syncs on CUDA (twice per leg) -- negligible at a quiesced flip seam, recorded so a profiler does not rediscover it as a mystery. Tests: 8 passed (unchanged), test_mamba_slot_union_801.py untouched.

…the order does not matter sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small pinned share (a "hot head, in wave order"), the rest file-backed as today, and a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art gate falsified two of those three, and the third rests on a comparison nobody has run. Recording that before writing code, because the cheapest outcome here is the one where nothing is rebuilt. THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed 2026-08-22, weights_arena.py:522) reads the file-backed image with bounded preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and a synchronize before a buffer is refilled. The depth is SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer. Rebuilding it would be the sgl-project#720-verbatim mistake in a new place. THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then checksums it; nothing reads the arena before that checksum passes. There is no partial publication and no per-wave consumption, so WHICH bytes are pinned cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward consumes experts wave by wave; a flip consumes the arena once, whole. A "hot head selection" would be a knob with no effect. TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the next reader does not "find" them by accident (1763 is a byte budget in an affordability check and an NVML corridor floor; 1844 is a seam-staging free memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS = 4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say: pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/ 3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x -- but from two separate campaigns, never against each other, and the note says so rather than presenting it as an A/B. COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse write records a 0 byte win on /spinning because ZFS folds the same holes -- which is where the flip images live. Independently, at ratio 1.145 the bytes saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64 GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks. The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars are out of reach by 2-13x. THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a swapless boot, and it refuses a missing or tmpfs image dir rather than "silently allocating a pinned image the host ledger would then double-count as reclaimable". The tree has ALREADY decided that image posts are registered but not checked -- "a new refusal path here could break a boot that works today", weights_arena.py:918-924. That decision stands. It does not bind a new, opt-in, default-off pinned share, for its own stated reason: refusing a share no current boot requests cannot break a boot that works today. So only the new share may be checked, and a hybrid image must register exactly its pinned head and nothing else, because the file-backed bytes are deliberately unregistered (the registry sums NON-reclaimable bytes). THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent: _staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression with no error and no log line -- and the pinned head costs more than it saves. The head must be floored to _DIRECT_ALIGN. NOT BUILT HERE, deliberately: the share's size is the whole feature, and the comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same load, same bytes) has never been run. A size derived from the two-campaign mixture above would be the rig-fit the planner rules forbid, and shipping the actuator with the size left as an unmeasured parameter would put a knob in the tree nobody can set -- while a helper with no caller is exactly the defect removed from planner/hicache_staging.py in this same branch. The A/B is named as a window item, with the instrumentation that already exists for it (_timed_arena_refill / refill_report). Analysis only: no code changes, no test changes, no behaviour change. codespell clean.

The user asked why a tp_to_pp flip takes 11.6 s, and whether 16 waves over 116502 live slots could not just be moved in one parallel shot. The capture answers the first half and refutes the premise of the second. THE 116502 LIVE SLOTS AND THE 16 WAVES ARE NOT THE COST. W25's own seam census names one segment: seam-census] timing tp_to_pp rank 0: 10466.8 ms across 448 segment(s), worst 'refill_highwater->weights_refill' 9516.2 ms (91% of the walk) kv_write->gdn_state 174.3 | cutover->done 152.4 | flip_writeback->hicache_quiesce 74.8 | kv_pack->kv_local_read 57.4 Three consecutive tp_to_pp flips: 9516.2 / 9496.5 / 12108.2 ms in that one segment, against totals of 10466.8 / 10568.0 / 13181.2. Meanwhile the ENTIRE KV movement at the largest observed reshard (epoch 11, 116502 live slots, 995.31 MiB sent / 825.03 MiB received) is read 19.9 + exchange 486.0 + write 394.8 = 901 ms, and the GDN blob is 11.69-18.70 MiB. Across the window live slots grew 947x (123 -> 116502) while the flip grew 1.26x (5077.9 -> 6416.0 ms). The ranks are already parallel: totals agree within ~40 ms while movers legs differ by ~600 ms, and the peers absorb PP0's overrun as CUTOVER WAIT (1313.3 / 1220.8 ms vs PP0's 225.3). A rendezvous, exactly as sgl-project#690 recorded. SO THE SEAM IS A WEIGHTS-REFILL RATE PROBLEM, and the rate is direction- asymmetric for reasons nothing in the tree explains. Same rank, within 2.7% of the same bytes: pp_to_tp 15925.8 MiB 3214-3915 MiB/s 4.07-4.96 s tp_to_pp 16362.7 MiB 1351-1723 MiB/s 9.50-12.11 s FOUR CANDIDATE EXPLANATIONS WERE CHECKED AND ALL FOUR FAIL, and they are recorded so nobody re-derives them: * not a path fallback -- SGLANG_PHASE_FLIP_REFILL_STAGED defaults True (environ.py:348), unset in the boot, and arena_refill dispatches on the image being file-backed, not on direction. Both take _staged_file_refill. * not a missing fd -- both sgl-project#802 warning paths ("could not open a read fd", "O_DIRECT unavailable") appear ZERO times in the 3.45 MB capture, against 9 "flip host image FILE-BACKED" registrations. * not the O_DIRECT alignment cliff (sgl-project#809's 8304 -> 2595 MiB/s) -- the loop only issues aligned offsets: chunks are 32 MiB multiples of _DIRECT_ALIGN=4096 and `want` is rounded down to it; only the trailing checksum tail is buffered, by design. * not the pre-sgl-project#802 fault path -- on sgl-project#802's own discriminator the fault path makes rank rates CONVERGE (821/775 MiB/s on links differing 1.80x). W25 DIVERGES with the link in both directions (1.59x and 1.40x). THE INSTRUMENT IS WHAT IS MISSING, NOT THE MECHANISM. The leg reports ONE aggregate MiB/s. The read (os.preadv) and the H2D (copy_ on a stream, depth 2) are pipelined, so that aggregate is min(read_rate, h2d_rate) with no way to see which bound it hit. That is the sgl-project#851 class -- one number with several meanings -- sitting inside the term that is 91% of the seam, which is why two independent readers could not attribute the gap from code. WHAT SHIPPED. `RefillLegTiming` accumulates, on the existing path: read_s wall time inside preadv -- storage/ARC bound h2d_wait_s wall time blocked on a prior DMA -- PCIe/link bound drain_s the pipeline tail, counted apart so it can never be mistaken for either bound They are near-exclusive by construction: the ring only waits on a buffer whose copy has not landed, so a read-bound leg never blocks there and a link-bound leg blocks almost every turn. `refill_bound_phrase` is a PURE function over that record -- no GPU -- so both directions are falsifiable off metal, the same split sgl-project#852 used for the allocator-cache estimator and for the same reason: a rule that can only be exercised on metal is one this corpus has repeatedly shipped inert. THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned. A phrase that always named a bound would satisfy every "it says something" assertion while being exactly as useless as the aggregate it replaces. So "unattributed" (not instrumented, or no time accounted) and "MIXED" (neither half dominates) are first-class outcomes, asserted directly, and a 101-point sweep across the whole read-share range proves the three verdicts neither overlap nor leave a hole. NO BEHAVIOUR CHANGE: the timing record is optional and defaults to None, so every caller that does not pass one runs byte-identically. ONE PINNING TEST HAD TO MOVE, and it is worth naming because widening it blindly was the tempting wrong answer. `test_arena_high_water_631.py`'s `fake_arena_refill` stub replaces `arena_refill` to prove the arena is committed before it is copied into; it did not accept the new kwarg and all six of its refill tests failed with TypeError -- the sgl-project#624 stub-drift shape. The stub now accepts `timing=None` (which is the whole adaptation, since the record is an instrument with no semantics the test pins), AND it captures what it was handed, with a new `test_the_refill_leg_is_instrumented` asserting the caller really does pass a `RefillLegTiming`. Without that, the signature widening would have silently tolerated the instrument being unwired again -- a stub that accepts anything proves nothing. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_refill_bound_856.py 9 passed, 101 subtests test/registered/unit/model_executor/ 15 failed / 763 passed, against a VERIFIED HEAD baseline of 15 failed / 754 passed -- identical failure set, pre-existing, unrelated to this path (bf16 vendor fallback, coresidence budget, expert-offload kv regain). Checked by reverting this change and re-running, not asserted. ruff format clean; the one F401 in phase_flip_boot.py is pre-existing at HEAD (verified against git show HEAD:...). Full ledger, with every figure and every failed search set: docs/dev/NOTE_856_seam_cost_ledger.md

feat: add chat template for internlm2-chat

da725d0

zhyncs requested review from Ying1123, hnyls2002 and merrymercy July 29, 2024 15:40

zhyncs self-assigned this Jul 29, 2024

Ying1123 approved these changes Jul 29, 2024

View reviewed changes

zhyncs merged commit 3d1cb0a into sgl-project:main Jul 29, 2024

zhyncs deleted the intern branch July 29, 2024 19:18

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

feat: add chat template for internlm2-chat (sgl-project#802)

b536088

amote-i pushed a commit to amote-i/sglang that referenced this pull request Dec 8, 2025

Update pre-downloaded-models.ini (sgl-project#802)

2144dbd

Navigation Menu

Uh oh!

feat: add chat template for internlm2-chat - #802

feat: add chat template for internlm2-chat#802
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:intern

zhyncs commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zhyncs commented Jul 29, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants