Repository navigation
[Feature] Support --tokenizer-worker-num > 1 in the offline Engine API - #40260
Merged
ishandhanani merged 3 commits intoSep 22, 2026
Merged
Conversation
Collaborator
|
/tag-and-rerun-ci |
ishandhanani
marked this pull request as ready for review
September 18, 2026 23:57
ishandhanani
requested review from
CatherineSue,
JustinTong0323,
ispobock,
merrymercy and
slin1237
as code owners
September 18, 2026 23:57
peilii
force-pushed
the
peili/engine-multi-tokenizer
branch
from
September 21, 2026 18:38
f8308d0 to
3c3b938
Compare
With tokenizer_worker_num > 1, Engine.tokenizer_manager was a MultiTokenizerRouter, which has no generate_request, so Engine.generate()/async_generate() failed at the first call (sgl-project#15157). The router only bridges TokenizerWorkers to the scheduler; the HTTP server side-steps this by running each uvicorn worker as a TokenizerWorker initialized from shared memory. Give the Engine API the same shape: - Engine.__init__ publishes (port_args, server_args, scheduler_info) to shared memory (write_data_for_multi_tokenizer) and creates an in-process TokenizerWorker, so the parent Engine keeps working; the router is kept as Engine.tokenizer_router. - Engine.attach_tokenizer_worker(parent_pid=None) builds an Engine in another process that owns only a TokenizerWorker registered with the parent's router: generate() runs there while prefill/decode stay on the parent's schedulers. This lets callers spread the per-request Python work of one engine over several processes. - shutdown(): attached engines only close their worker; the parent unlinks the shm. Fixes sgl-project#15157
…gisters TokenizerWorker.__init__ sends its registration to the router over a zmq.asyncio socket, which schedules on the current event loop. Engine only created its loop after the tokenizer worker, so a plain sgl.Engine(tokenizer_worker_num=2) from a thread without a loop (the CI test) failed with 'There is no current event loop'. Establish the loop first, on both the constructor and attach_tokenizer_worker paths, and use the same loop object for generate().
peilii
force-pushed
the
peili/engine-multi-tokenizer
branch
from
September 21, 2026 21:35
3c3b938 to
8c317fa
Compare
Collaborator
|
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With
tokenizer_worker_num > 1,Engine.tokenizer_manageris aMultiTokenizerRouter, which has nogenerate_request, soEngine.generate()/async_generate()fail on the first call (#15157). The router only bridgesTokenizerWorkers to the scheduler. The HTTP server side-steps this by running each uvicorn worker as aTokenizerWorkerinitialized from shared memory, but the offline Engine API has no equivalent, so embedders such as Dynamo (ai-dynamo/dynamo#15078) cannot spread the per-request Python work of one engine over several processes.Modifications
Give the Engine API the same shape as the HTTP server:
Engine.__init__: whentokenizer_worker_num > 1, publish(port_args, server_args, scheduler_info)to shared memory via the existingwrite_data_for_multi_tokenizer, and create an in-processTokenizerWorkerso the parent Engine'sgenerate()keeps working. The router is kept asEngine.tokenizer_router.Engine.attach_tokenizer_worker(parent_pid=None)(new classmethod): build an Engine in another process that owns only aTokenizerWorkerregistered with the parent's router.generate()runs there while prefill/decode stay on the parent's schedulers.Engine.shutdown(): attached engines only close their worker; the parent unlinks the shared memory.No behavior change for
tokenizer_worker_num == 1.Fixes #15157
Accuracy Tests
Not applicable: no change to model forward or kernels. The new test checks that an attached process returns the same text and token counts as the parent for greedy decoding.
Speed Tests and Profiling
Not applicable to single-process use; the change is opt-in via
tokenizer_worker_num.Checklist
test/registered/core/test_engine_multi_tokenizer.py, registered for CUDA CI (1 GPU).🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #35658106558
Latest PR Test (Extra): ❌ Run #35658106054
Latest PR Test (AMD ROCm 10): ❌ Run #35658106548