Add skip_tokenizer_init args. by gryffindor-rr · Pull Request #959 · sgl-project/sglang

gryffindor-rr · 2024-08-07T04:40:47Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

#924

Modification

Added a skip_tokenizer_init to skip tokenizer and use the input_ids directly in tokenizer manager.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

zhyncs

Hi @gryffindor-rr May you add a unit test for this? Thanks.

Add unittest for srt with skip_tokenizer_init.

zhyncs · 2024-08-07T11:29:14Z

Hi @gryffindor-rr I've update the deps for SGLang, may you merge the latest main? Thanks.

Ying1123

Can you run your code successfully?

I think you need also to change this part

sglang/python/sglang/srt/managers/tokenizer_manager.py

Lines 484 to 486 in db6e26d

    
           recv_obj: BatchTokenIDOut = await self.recv_from_detokenizer.recv_pyobj() 
        
           assert isinstance(recv_obj, BatchStrOut)

.

Please fix the lint as well.

Passed with set the model_path to my local model.

gryffindor-rr · 2024-08-08T09:48:26Z

Hi @gryffindor-rr I've update the deps for SGLang, may you merge the latest main? Thanks.

done.

merrymercy

could you fix the lint errors? https://github.com/sgl-project/sglang/blob/main/docs/en/contributor_guide.md#format-your-code

zhyncs · 2024-08-09T12:20:25Z

@gryffindor-rr Please resolve the CI issue.

gryffindor-rr · 2024-08-09T12:28:01Z

could you fix the lint errors? https://github.com/sgl-project/sglang/blob/main/docs/en/contributor_guide.md#format-your-code

done.

gryffindor-rr · 2024-08-09T13:10:17Z

@gryffindor-rr Please resolve the CI issue.

done.

merrymercy

Great work!

merrymercy · 2024-10-26T00:29:31Z

The tokens returned by this implementation is wrong. It is fixed by #1798

Co-authored-by: lzhang <zhanglei@modelbest.cn>

GPU case transplantation

…arithmetic not a check WHAT DIED. window-955-boot's second boot (pin 27bcb48, boot_943bx_27bcb4884f_0828_025835.log) took `assert self.chunked_req is None` in `_get_new_batch_prefill_raw` on ALL THREE ranks at 03:03:29 -- three seconds after `PHASE-FLIP DONE pp_to_tp (epoch 1) in 9709.5 ms`, the first clean cutover this family has produced, with 9 phase=tp batches already run. The line immediately before the crash on every rank: `PHASE-FLIP armed (tp_to_pp) but NOT QUIESCENT: a chunked prefill is incomplete` -- a continuation was demonstrably resident when a second one was minted. NOT A sgl-project#951 REGRESSION, and sgl-project#951 said so itself: its comment at the assert records that the invariant is held by ARITHMETIC (a surviving continuation has normally spent all of rem_chunk_tokens, so the fresh branch computes trunc_len <= 0), that this is BREAKABLE with witnesses, and that sgl-project#951 closes only the PP instance -- "It does NOT close the general case, which needs its own posten and its own danger-direction analysis." AND ITS GREEN WAS VACUOUS. window-951 read 0/0 on this line while every batch it saw was phase=pp. Boot 2 is the first evidence the TP side is reachable at all, which is why the new suite proves reachability before asserting anything. WHY THE EXISTING CLEAR DOES NOT COVER IT. phase_flip_runtime.py:1853-1856 clears `scheduler.chunked_req` at the seam only for a request whose id() is in the RETRACTED target set. A continuation that survives the flip un-retracted -- the designed behaviour, see `chunk_blocks_quiescence` -- is never in that set. A cutover also resizes the pool, which is precisely the mid-pass replenishment witness_941_a needs to mint the second continuation. DANGER DIRECTION, and it decides where the fix goes. The two ways to restore the invariant are not symmetric: * refuse the FRESH chunked admission -- nothing of it is committed, no KV is held, no chunk has run. It waits one pass, via the requeue-for-free the admission loop already relies on. Nothing is lost, so no double prefill. It cannot starve: the resident continuation is consuming chunks, and when it finishes the fresh one is admitted. * drop the RESIDENT continuation (clear chunked_req at the cutover) -- that re-prefills a request mid-flight. That is the double prefill the standing law forbids outright, and the sgl-project#858 wedge shape besides. The resident continuation is never the one to give way. FIX. `PrefillAdder.chunked_req_outstanding`, stamped by the scheduler at the one point where residency is settled -- right after the add_chunked_req branch, which matters because the sgl-project#906 seam refusal KEEPS the continuation without calling the adder at all, so residency cannot be inferred from the adder's own calls. The two fresh-request mint sites in `add_one_req` return AddReqResult.OTHER while it is set. No new mechanism: `_add_scheduled_req`'s `carried_chunk` flag is the precedent and already names this assert while refusing for the same reason. Default False, so any adder built outside the scheduler behaves exactly as before. THE ASSERT STAYS. It is the honest watcher and has now named its own reachability twice; nothing here weakens it. TESTS. test_second_chunked_req_959.py, 7 tests, driving the REAL PrefillAdder through add_chunked_req / preempt_to_schedule / add_one_req -- adapted from /spinning/evidence-665-f1/witness_951/witness_941_a.py rather than inventing a second harness. Includes the reachability proof (the continuation really does survive with chunk budget left), a CANFAIL arm showing the unguarded adder reproduces the crash precondition (both a resident and a freshly minted continuation), that the resident one keeps its committed geometry byte for byte (kein Doppel-Prefill), and that the guard does NOT fire when nothing is resident, so the fix costs no throughput. Desk gate, /spinning/htsglang-gpu/.venv, CVD="": BEFORE (pin) serial 878 passed / 2 failed AFTER (sgl-project#958+sgl-project#959) serial 895 passed / 2 failed; wide 3701; narrow 202 +17 = exactly the 10 tests of sgl-project#958 plus these 7. Failure set IDENTICAL, both the known pre-existing test_collective_family_siblings_610.py pair. Tally gate OK on all three lanes. 88/88 across the 946/951/955/958/959 suites. No boot. Metal proof belongs to the window ticket.

…te the group, and let the guard speak sgl-project#965 -- THE CO-DERIVED MATCH TUPLE. `Req.init_next_round_input` reads the radix tree ONCE and unpacks that single `match_result` into EIGHT attributes in one tuple assignment: prefix_indices, last_node, last_host_node, best_match_node, host_hit_length, swa_host_hit_length, mamba_host_hit_length, mamba_branching_seqlen. One reading of one geometry wearing eight names. `truncate_prefix_to` moves the prefix they were derived from. Its own docstring says why that matters -- "the geometry was DERIVED from the prefix that just moved, so it is now a stale reading rather than a report" -- and that sentence is true, word for word, of every field in the tuple. It wrote prefix_indices, and over two separate windows, each discovered by a boot, cache_protected_len (sgl-project#930) and extend_range (sgl-project#958). Six were left standing. This is the complete sibling sweep those two never got. NOTHING RE-DERIVES IN BETWEEN. scheduler.py calls init_next_round_input (all eight derived), both arms of the `pp_size > 1` fork call truncate_prefix_to, then adder.add_one_req reads them. No match_prefix on that path. WHAT THE STALE READINGS DO, at PrefillAdder.add_one_req: * `real_input_tokens = cand_extend_input_len - req.host_hit_length` subtracts a host hit no longer part of this prefix, under-counting against the budget; * `needs_host_load_back()` is still true, so init_load_back runs on a stale best_match_node/host_hit_length and does `prefix_indices = torch.cat([prefix_indices, new_indices])` -- leaving `[0, told)` then `[L_dev, L_dev+H)` with a HOLE between, while prepare_for_extend sizes the cross-stage tensor off len(prefix_indices) as though it were contiguous. The silently-wrong-context class; * that branch then sets `cache_protected_len = prefix_len`, re-raising the exact value the truncation lowered nine lines earlier -- undoing sgl-project#930. FIXED AS ONE CONTRACT, not a seventh one-field patch, because two of the eight were already bought one at a time by two boots. The ratchet (`test_every_co_derived_field_is_accounted_for`) reads the producer tuples from the AST -- not by grep, since a grep needs a name and the whole point is a field nobody has thought of yet -- and FAILS when a ninth joins, telling the author to decide TRUNCATED / CLEARED / HANDLE. Planted-ninth-field proof: it names `newly_added_ninth_reading`. `last_node` IS NOT NULLED, and that asymmetry is deliberate: it is not a reading but a RESOURCE HANDLE holding an outstanding inc_lock_ref, and this attribute is the only surviving reference to it. Nulling it would leak the ref and make the node permanently unevictable -- a defect that already exists on the PP void path (`_release_dynamic_chunk_probe` frees rows without cache_finished_req / dec_lock_ref, unlike schedule_batch.py:2170-2175 and disaggregation/prefill.py :1159-1161, which do). Registered, NOT fixed here. A NO-OP TRUNCATION STILL INVALIDATES NOTHING. My first test asserted the opposite; the invalidator's own documented rule -- "a no-op truncation leaves a valid geometry valid, clearing it there would void healthy passes for nothing" -- is right, and clearing there would discard a host hit every rank still holds. The test was inverted to pin the documented behaviour instead of contradicting it. sgl-project#967 -- THE sgl-project#959 GUARD WAS UNMEASURABLE. It is closed by two bare `return AddReqResult.OTHER` statements. A refusal that leaves no trace is indistinguishable from a scheduler that simply built nothing, which is precisely the state the sgl-project#963 investigation spent a boot separating by hand out of per-rank coverage databases. Unconditional per-site counter plus a rate-limited `[sgl-project#967] SECOND CONTINUATION REFUSED` line, following the neighbour that already got this right (`_note_seam_chunk_refused`: first three, then every thousandth). Module-level, because PrefillAdder is rebuilt every pass and an instance counter would reset before anyone could read it. THE TWO GUARDS ARE IN DIFFERENT METHODS -- add_one_req_ignore_eos:1692 and add_one_req:1935 -- and my first wiring pin inspected only add_one_req, so it measured nothing and failed on the UNMUTATED tree. That is how the mistake was caught. Rewritten to walk the module AST, and it now names the offending line number when a guard's counter is stripped (proof: "guard(s) at line(s) [1935] return without counting"). The site labels are the method names. CAN-FAIL, read rather than asserted: sgl-project#965 clearing withdrawn -> 8 failed, one per field, each naming its own field sgl-project#965 ninth field planted -> ratchet names it sgl-project#967 one guard's note stripped -> wiring pin names line 1935 NUMBERS, scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), 580.49 s, 358 modules count probe 2 named == 2 summary, SUBFAILED and ERRORS included truncation family + prefill adder + schedule_policy suites: 54 passed ruff: schedule_batch.py 45 before / 45 after, schedule_policy.py 21 / 21, both new test files clean TICKET_961_WINDOW.md gains section 8: the "race" of section 7 is NOT a race (ServerArgs byte-identical, the LOAD ARM differed -- 8422 chunked vs 2822 sub-chunk, and an over-one-chunk prompt forces a chunked_req onto every rank, so Arm A masks the defect), and the acceptance now counts flip EPOCHS and phase=tp BATCHES SEPARATELY: 3+3 "both directions" was 3 RANKS confirming ONE epoch, and no batch ever ran phase=tp, so every post-flip guard's green was vacuous. NOT CLAIMED: no boot, no metal, no effect measurement.

…resident at all Boot 14 (cf16281) died on PP1 after 39 s: `assert self.chunked_req is None`, scheduler.py:9784 in `_get_new_batch_prefill_raw`. Third recorded fundstelle of this family after :9286 (sgl-project#951) and :9367 (sgl-project#959). ROOT, and it corrects my own attribution in the boot-14 register entry. The invariant behind that assert is held "by ARITHMETIC, not by a check" (scheduler.py's own comment). sgl-project#959 therefore gave the two `add_one_req*` mint sites an explicit `chunked_req_outstanding` check, and SKIPPED the third -- `PrefillAdder._add_scheduled_req`, the forwarded-schedule execution path -- reasoning at schedule_policy.py that it "already has its own (`carried_chunk`)". That is the guard-comment-names-the-hazard trap. `carried_chunk` answers "is THIS request the resident continuation". The invariant needs "is there a resident continuation AT ALL". It covers a request being re-announced; it does not cover a DIFFERENT named request becoming a second continuation while the first is resident. Two of three sites guarded, one not. Boot 14 is that gap on metal: the resident continuation survived `add_chunked_req`, so `chunked_req_outstanding` was True and both sibling sites correctly refused -- and this site minted anyway, on another rid the same forwarded schedule named. sgl-project#994 EXPOSED THIS, IT DID NOT CREATE IT. Boot 13 never reached the line because the sgl-project#791 geometry refusal killed every pass before a batch was built; sgl-project#994 removed that refusal. My register entry attributed the death to sgl-project#994's effect on `rem_chunk_tokens` arithmetic. That was wrong: both sibling sites were already guarded and did refuse. The arithmetic was not the hole; the missing third guard was. Withdrawn here rather than left standing. FIX: the sibling guard, at the site that lacked it. WHY A PASS REFUSAL AND NOT A REQUEST SKIP -- the danger direction, which is the whole question here. On a forwarded schedule this rank may NOT drop a named request: the upstream's hidden states for it are already on the wire, which is what this same method already raises `PPScheduleRefused` for a few lines above. Running the chunk WITHOUT announcing it is worse -- the continuation would be untracked and re-prefilled next pass, the double prefill the standing law forbids. So the disposal is the one this path already owns: refuse the PASS by name, let sgl-project#791/sgl-project#797 void and re-derive. It cannot starve: the resident continuation is consuming chunks, and when it finishes `chunked_req` is None and the schedule is executable. This is the same direction sgl-project#959 chose ("the resident continuation is never the one to give way; the fresh admission is") -- not the sgl-project#858 mid-prefill wedge, which would be clearing `scheduler.chunked_req` instead. FUTURE CHECK: `grep -c 'if self.chunked_req_outstanding:'` over schedule_policy.py is now 3 and equals the number of `new_chunked_req` writers. A fourth writer that skips it is visible as an inequality rather than as a boot death. Execution proof (speed mode, one instrument per link): `note_second_continuation_refused(req, "_add_scheduled_req")` counts it and the refusal text names itself, so boot 15 measures whether the guard fires at all -- absence of the assert alone would be green-by-absence. Evidence: desk. py_compile; local-import dominance and guard-before-raise- before-mint verified by source order in the loaded module; guarded-site count 3 == mint-site count 3. Belegstufe: DESK-BEWIESEN.

…the pass Withdraws sgl-project#995's `PPScheduleRefused`. The guard was right, the disposal was wrong, and boot 15 measured it. BOOT 15 (473f3ad, 22:12-22:13): 175 `sgl-project#995 FORWARDED SCHEDULE UNEXECUTABLE` on exactly ONE rid (ddb6f38b…) in ~40 s, 4 batches total, rank 2 at 0% GPU while ranks 0-1 burned on corridor-reclaim. That is a LIVELOCK, not a stall: a voided pass computes nothing, so the resident continuation never advanced and each following pass rebuilt the identical refusal. It is the same shape this branch already records as "Boot 3 spent 514 refusals on that asymmetry". MY OWN STARVATION ARGUMENT WAS THE DEFECT. sgl-project#995's comment claimed "it cannot starve -- the resident continuation is consuming chunks". I took that sentence from sgl-project#959, where it is true because that path SKIPS a request and lets the pass run. On a pass REFUSAL nothing runs, so the premise is false. Borrowing a justification across paths without a coverage check is the guard-comment class this window has now produced twice. SECOND REASON, independent: the `except PPScheduleRefused` handler in scheduler.py documents a deliberately-open lock_ref leak (requests admitted earlier in the same loop keep their `inc_lock_ref` when the batch never completes) and justifies leaving it open with "Bounded: it takes a genuinely unexecutable geometry to reach this line at all". This condition is ordinary, not exotic, so the refusal converted a bounded leak into a per-pass ratchet: protected nodes, falling evictable_size, a slow wedge instead of a crash. WHY NOT ANNOUNCING LOSES NOTHING, which is what the raise got wrong. `new_chunked_req` / `scheduler.chunked_req` is LOCAL bookkeeping for a chunk this rank decided itself, so the next pass can find and resume it. Under a FORWARDED schedule that job is the upstream's: `_add_scheduled_req` receives `prefix_len` and `extend_len` fresh every pass (its own "LAST CHUNK OR NOT IS ALSO THE SCHEDULE'S TO SAY"), so the continuation is re-established from the decision whether or not this rank remembered it. The chunk runs as decided. No dropped named request, no re-prefill, and the single field keeps its one occupant so the assert stays intact. The guard itself stands: three `new_chunked_req` writers, three `chunked_req_outstanding` checks. `note_second_continuation_refused(req, "_add_scheduled_req")` still counts every occurrence, so the condition stays visible without costing a pass. Credit: the refusal direction was challenged by a desk agent on control-flow reading (lock-ref leak reachability and the livelock) before boot 15 confirmed both. Recorded because the challenge was right and the metal agreed with it, not with me. Evidence: desk + the boot-15 metal specimen that condemns the previous version. py_compile; guard-before-mint order and refusal-text absence verified in the loaded module; guarded-site count 3 == mint-site count 3. Belegstufe: sgl-project#995's refusal METALL-WIDERLEGT; this replacement DESK-BEWIESEN.

Boot 16 (996fbf4, 22:21:48, boot_943bx_996fbf4aca_0828_221614.log) died 68 s after first load. PP1 and PP2 both logged sgl-project#987 FILL-ADOPT rid=da614e20... local=8446 -> upstream=8447 appended=1 tail=[271] and PP1 then took scheduler_pp_mixin.py:2147 RuntimeError: sgl-project#631 PROXY LEFTOVER REFUSED: a proxy stamped mb_id=1 seq=17 rows=4096 epoch=2 arrived while this rank is on mb_id=2 in flip epoch 2 -- the SAME-epoch branch, i.e. a proxy for a pass this rank had already left. All three ranks down. THE ROOT IS NOT WHERE sgl-project#995 LOOKED. `_add_scheduled_req` decided whether a forwarded extent finishes its request with last_chunk = prefix_len + extend_len >= local_fill_len under a comment claiming that was "the schedule's to say ... arithmetic on forwarded integers rather than re-taken against a local budget". Two of the three integers are forwarded. The third is not: `local_fill_len` is `len(full_untruncated_fill_ids)`, rebuilt from THIS rank's origin + output_ids (+ carried tail) on every pass by `Req._refresh_fill_ids` (schedule_batch.py:1326 -- unconditional, ahead of the tree_cache gate at :1355). The verdict was a rank-local quantity wearing a forwarded one's clothes, and the fill is exactly the quantity this seam is known to disagree about. THE GUARD COVERED ONLY HALF THE DISAGREEMENT. `schedule_refusal_reason`'s third clause refuses `prefix + extend > local_fill_len` -- the decision asking for more than this rank holds. The opposite skew, this rank holding MORE than the rank that decided, passes every clause and silently flips "last chunk" into "mint a continuation". `adopt_carried_fill` cannot close it either: it only APPENDS (:1628-1633), so it lifts a short follower up to the decider and never brings a long one down. 8446 vs 8447 is that skew, and the continuation nobody decided on is what put the leftover proxy on the wire. THE FIX IS A CARRY, NOT A FOURTH CLAUSE. `PPAdmissionEntry` gains `last_chunk`, set by the deciding rank in `build_pp_admission_decision` from the same `fill_carry_for` reader that already publishes `fill_len`, projected by `forwarded_last_chunk` (the third projection of the one decision object, alongside `forwarded_fill_carry` and `_pp_forwarded_schedule_from`, so all three empty on the same passes and can never name different rid sets), and executed at the receiver. Tightening the refusal clause to `!=` was the available alternative and is wrong: `prefix + extend < local_fill_len` is the normal case for every legitimate middle chunk, so that refuses every pass. BOTH READERS OF THE VERDICT MOVE TOGETHER. `add_chunked_req` asks the same question one method later ("is this still the chunked request") and answered it by reading `extend_range.end` back off the request -- equivalent only while BOTH sides derived it locally. It now reads the same carried source via `_told_last_chunk`, or the two halves of one fact would drift, which is this ticket's own defect class one level up. `None` MEANS SAY NOTHING. A legacy sender, an unreadable fill, PP0 itself and every pp_size<=1 boot all yield no entry in the map, and both readers fall back to the pre-sgl-project#996 local derivation unchanged. RATCHET, same diff: the three `new_chunked_req = req` sites were bare assignments, so a second mint in one pass would have SILENTLY overwritten the first -- the overwritten request left in can_run_list with a partial extend range, tracked by nobody, re-prefilled next pass (the double prefill the standing law forbids, with no assert anywhere on the path). They now go through `_mint_chunked`, which asserts `new_chunked_req is None` and names both rids and the site. Nothing has been observed doing this; `chunked_req_outstanding` is set once per pass and knows only about the RESIDENT continuation, so neither existing watcher could see it. This is the check the family (sgl-project#951, sgl-project#959, sgl-project#995, sgl-project#996) kept discovering it did not have. DESK EVIDENCE, the boot-16 skew reproduced end to end and falsified in the dangerous direction: producer holding 8446 with a chunk landing on its end publishes last_chunk=True; a receiver holding 8447 derives False locally and would mint, and executes True with the carry. Old behaviour mints, new does not. Import smoke on all three modules; ruff clean on all 265 added lines (the 125 findings in these files are pre-existing and untouched). NOT yet boot-proven -- that is the next boot's job, and the sgl-project#996 log line fires exactly when the carry changes the verdict so the seam is readable directly.

The arithmetic axis withdrew its causal claim and supplied two facts with it. Both re-run against the log here, because the rule that agent reports are not evidence applies to a retraction as much as to a finding. NK-1: grep -c AssertionError over the whole Boot 16 log is 0. The `assert self.chunked_req is None` never fired. That assert is Boot 14's death; Boot 16 is a different one, and one narrative was covering two. NK-2, and it is stronger than the ordering argument it replaces: SECOND CONTINUATION REFUSED appears exactly twice, :2034 (22:20:45 PP1) and :2045 (22:20:46 PP2) -- BEFORE the armed window opens at 22:20:47. The sgl-project#959/#995b guard held and no second continuation was ever minted. A path that is guarded, handled, and finished before the window cannot kill after it. So the claimed producer did not exist and its characteristic death did not occur. The ring divergence stands as Boot 16's root. Two predictions now stand side by side for the next boot under Cut C, on disjoint quantities so one boot answers both: this axis says the sgl-project#631 leftover SURVIVES (slot index, hangs on the chunk asymmetry at :5330-5331); the arithmetic axis says the two SECOND CONTINUATION REFUSED go to ZERO (mint counter). Both landing means two defects were separated rather than one told twice. And the class is the same at both seams, which is their observation and a good one: schedule_policy.py:1433 derives last_chunk from a rank-local fill length, :5330-5331 exempts the hold on a rank-local chunked_req -- a rank-local predicate deciding a quantity that must be group-uniform. That is Family B of ANALYSE_996, found twice independently.

…a dict the cut swallowed TWO EMITTERS MADE HONEST, both instances of the sampling trap this campaign has now hit four times. `sgl-project#988 LOADBACK` printed the FIRST event and then every 64th, with no suppression count. Three lines across three ranks were therefore indistinguishable from "one application per rank" and from "sixty-three per rank that were never printed" -- and a reproducibility run on boot 15 read exactly that ambiguity and could conclude NOTHING, in either direction. This is the quantity the whole slice is judged on, so it is now unconditional. `sgl-project#1042 EXTENT LIFECYCLE` sampled at n<=3 then every 256th. Boot 15's acceptance window contained ZERO such lines while the log held nine: the window fell between samples, and a reader would have concluded "the contract was never exercised". `set` and `retract` are rare and now print unconditionally; only `hitless_noop` is still sampled, and it prints how many it skipped. RUFF CAUGHT A REAL BUG, not a style nit: `_SECOND_CONTINUATION_REFUSALS` (F821) was swallowed by the sgl-project#1046 cut, which deleted the old `_pp_load_back_extent` body by slicing to the next `def` -- and that module-level line sat inside the span. The NameError would have fired only on the sgl-project#959 second-continuation refusal path, i.e. under specific traffic and probably not on the next boot. Restored with a note. This is the argument for keeping ruff in the loop even under speed mode: it is not a suite, it is one pass that can fail on exactly the class a text-slice deletion produces. check_1042 and check_1045 re-run green; import smoke confirms the restored ledger.

lzhang added 2 commits August 7, 2024 12:02

Add skip_tokenizer_init args.

798b319

Fix wrong clause.

670a374

zhyncs requested review from Ying1123, hnyls2002, merrymercy, yichuan-w and zhyncs August 7, 2024 05:02

gryffindor-rr mentioned this pull request Aug 7, 2024

[Feature] tokenizer_manager accept external tokenizer or skip tokenizer init #924

Closed

zhyncs reviewed Aug 7, 2024

View reviewed changes

Add skip_tokenizer_init to detokenizer manager.

db6e26d

Add unittest for srt with skip_tokenizer_init.

Ying1123 requested changes Aug 8, 2024

View reviewed changes

Comment thread python/sglang/srt/managers/detokenizer_manager.py Outdated

Comment thread test/lang/test_skip_tokenizer_srt.py Outdated

Comment thread test/lang/test_skip_tokenizer_srt.py Outdated

Merge branch 'main' of github.com:sgl-project/sglang into feat/tokenizer

b6ce08d

merrymercy added the high priority label Aug 8, 2024

Add unittest to test skip_tokenizer_init.

3f62200

Passed with set the model_path to my local model.

merrymercy requested changes Aug 8, 2024

View reviewed changes

Comment thread python/sglang/srt/server.py Outdated

Comment thread python/sglang/srt/managers/tp_worker.py Outdated

Comment thread python/sglang/srt/managers/tokenizer_manager.py Outdated

lzhang added 2 commits August 9, 2024 20:15

Merge branch 'main' of github.com:sgl-project/sglang into feat/tokenizer

a4d996a

Merge branch 'main' of github.com:sgl-project/sglang into feat/tokenizer

71625d9

lzhang added 2 commits August 9, 2024 20:23

Fix running error. modify according to review.

75074c4

Fix lint error.

710f136

lzhang added 4 commits August 9, 2024 20:36

Fix test.

277a836

Fix testing.

85505e6

Fix lint error of skip tokenizer test.

2556644

Fix lint.

dc18a29

Merge branch 'main' into feat/tokenizer

a7fa37e

merrymercy approved these changes Aug 9, 2024

View reviewed changes

merrymercy enabled auto-merge (squash) August 9, 2024 19:10

Ying1123 approved these changes Aug 9, 2024

View reviewed changes

merrymercy merged commit 9cf0a5b into sgl-project:main Aug 9, 2024

zhyncs mentioned this pull request Aug 14, 2024

[Feature] modify Runtime to support skip_tokenizer_init #1088

Merged

merrymercy mentioned this pull request Oct 26, 2024

[Fix] Fix --skip-tokenizer-init #1798

Merged

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Add skip_tokenizer_init args. (sgl-project#959)

27d9444

Co-authored-by: lzhang <zhanglei@modelbest.cn>

qybnb pushed a commit to qybnb/sglang that referenced this pull request Jul 10, 2026

Merge pull request sgl-project#959 from A6444/sglang_l3005

2b9b943

GPU case transplantation

Navigation Menu

Uh oh!

Add skip_tokenizer_init args. - #959

Add skip_tokenizer_init args.#959
merrymercy merged 14 commits into
sgl-project:mainfrom
gryffindor-rr:feat/tokenizer

gryffindor-rr commented Aug 7, 2024

Uh oh!

zhyncs left a comment

Uh oh!

zhyncs commented Aug 7, 2024

Uh oh!

Ying1123 left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gryffindor-rr commented Aug 8, 2024

Uh oh!

merrymercy left a comment •

edited by Ying1123

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhyncs commented Aug 9, 2024

Uh oh!

gryffindor-rr commented Aug 9, 2024

Uh oh!

gryffindor-rr commented Aug 9, 2024

Uh oh!

merrymercy left a comment

Uh oh!

merrymercy commented Oct 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	recv_obj: BatchTokenIDOut = await self.recv_from_detokenizer.recv_pyobj()
	assert isinstance(recv_obj, BatchStrOut)

Uh oh!

Conversation

gryffindor-rr commented Aug 7, 2024

Motivation

Modification

Checklist

Uh oh!

zhyncs left a comment

Choose a reason for hiding this comment

Uh oh!

zhyncs commented Aug 7, 2024

Uh oh!

Ying1123 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gryffindor-rr commented Aug 8, 2024

Uh oh!

merrymercy left a comment • edited by Ying1123 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhyncs commented Aug 9, 2024

Uh oh!

gryffindor-rr commented Aug 9, 2024

Uh oh!

gryffindor-rr commented Aug 9, 2024

Uh oh!

merrymercy left a comment

Choose a reason for hiding this comment

Uh oh!

merrymercy commented Oct 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Ying1123 left a comment •

edited

Loading

merrymercy left a comment •

edited by Ying1123

Loading