[anencore94]

Problem

Long regular HF fast-tokenizer calls in TokenizerManager._tokenize_texts() block the asyncio event loop and delay handlers such as /ping. Moving the call to a worker is insufficient if loop-side callers still share its native backend: a padding/truncation update can wait on the worker's encode.

Addresses the regular HF fast-tokenizer case in #30770. OpenAI conversion is already offloaded by merged #39771; this PR leaves that implementation in place.

Changes

• Give regular HF fast tokenization one worker and a separate native backend, cloned once at startup. Pass the tokenizer explicitly to the worker rather than swapping self.tokenizer or introducing context-dependent tokenizer properties.
• Shallow-copy the wrapper per call so later wrapper settings are reflected without copying the vocabulary on the request path. Propagate request contextvars.
• Preserve synchronous behavior for slow/custom tokenizers and a tokenizer whose backend has been replaced. Preserve the dynamic-batch branch, cross-encoder/result handling and existing OpenAI conversion parallelism.
• Add registered CPU coverage for loop and real ASGI /generate + /ping responsiveness, backend separation, current wrapper settings, fallback behavior, serialization and context propagation. HTTP tests substitute model execution.

The backend-isolation direction follows @chromecast56's work and @hnyls2002's shared-tokenizer review on #39716. #39716 and @JianWang97's #44502 both build on this PR; coordination to choose one merge path is open. This can remain a small prerequisite or be folded into #39716 with attribution preserved.

Validation

Base: e2982985110fb432fd8ddba5f0147a5004fd03ab.

• 388 passed, 376 subtests passed across the focused tests, request cleanup/configuration, chat, completions, embeddings, Responses, transcription and tokenize endpoints.
• Focused tests on the unmodified base: 2 expected starvation failures, 1 passed, 3 helper-specific skips. The candidate passes all 6 focused tests.
• Applicable pre-commit checks and git diff --check passed.
• Synthetic CPU WordLevel contention probe, actual candidate _tokenize_texts, tokenizers==0.23.2: a loop-side padding call during a 250k-token encode waited 296–307 ms with a deliberately shared-backend control, versus 0.20–0.33 ms with the candidate's separate backend (3 trials each; matching token outputs).

Local checks used macOS arm64 / Python 3.11.13, Transformers 5.19.0 and the repository's CPU kernel-stubbing helper. The contention probe is a synthetic correctness investigation, not a GPU or model-serving benchmark. Short-prompt bench_serving TTFT at low/high concurrency remains unvalidated. Upstream lint passed on e0a728540. Upstream test execution remains blocked on the missing run-ci label (current gate); maintainer review remains pending.

Scope and limits

This adds one native-backend copy per tokenizer manager. Runtime vocabulary mutation is outside this change; backend replacements retain the synchronous path. Slow/custom tokenizer offload, other endpoint preprocessing and global ordering/admission behavior remain separate work. There is no model, kernel, or KV-cache change.

Checklist

• Format code and run applicable pre-commit checks.
• Add registered CPU unit and HTTP regression coverage.
• Preserve the existing dynamic tokenizer and OpenAI conversion paths.
• No new user-facing configuration required.

---

CI States

Latest PR Test (Base): ❌ Run #41331127112
Latest PR Test (Extra): ❌ Run #41331127183
Latest PR Test (AMD ROCm 10): ❌ Run #41331127133

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[gemini-code-assist]

Code Review

This pull request improves event loop responsiveness by offloading potentially blocking operations, such as request conversion and tokenization, to separate threads using asyncio.to_thread. It also introduces unit tests to verify that these operations do not block the event loop. The review feedback highlights a critical thread-safety concern: slow tokenizers implemented in pure Python are not thread-safe and running them concurrently in separate threads can lead to race conditions or crashes. It is recommended to serialize access to slow tokenizers using a lock.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Offloading tokenization to a separate thread via asyncio.to_thread is a great improvement for event loop responsiveness. However, if the tokenizer is a slow tokenizer (i.e., not getattr(self.tokenizer, "is_fast", False)), it is implemented in pure Python and is generally not thread-safe. Running it concurrently in multiple threads can lead to race conditions, corrupted tokenization outputs, or crashes.

Furthermore, since pure Python code does not release the GIL, running multiple slow tokenizations concurrently in separate threads does not provide any CPU parallelization benefits anyway.

To ensure thread safety for slow tokenizers while keeping the asyncio event loop responsive, we should serialize calls to slow tokenizers using a lock (e.g., a threading.Lock initialized on the manager).

            if not hasattr(self, "_tokenizer_lock"):
                self._tokenizer_lock = threading.Lock()

            def tokenize_sync():
                is_fast = getattr(self.tokenizer, "is_fast", False)
                if not is_cross_encoder and not is_fast:
                    with self._tokenizer_lock:
                        return [self.tokenizer.encode(t) for t in tokenizer_input], None

                if not is_fast:
                    with self._tokenizer_lock:
                        encoded = self.tokenizer(tokenizer_input, **tokenizer_kwargs)
                else:
                    encoded = self.tokenizer(tokenizer_input, **tokenizer_kwargs)

                return (
                    encoded["input_ids"],
                    encoded.get("token_type_ids") if is_cross_encoder else None,
                )

            input_ids, token_type_ids = await asyncio.to_thread(tokenize_sync)

[anencore94]

Good catch. Fixed in edde3b7106adbc6080ffc94f391310f093039759. The follow-up replaces the default executor with a TokenizerManager-owned ThreadPoolExecutor(max_workers=1), shared by OpenAI request conversion and regular tokenization. This keeps preprocessing off the asyncio event loop while preserving the previous per-process serialization for slow tokenizers and shared template state. The helper also copies contextvars into the worker thread. I added regression coverage for serialization and context propagation; the related suite now passes 141 tests.

[anencore94]

@merrymercy @slin1237 This PR is ready for review. Could an authorized maintainer add the run-ci label and trigger the full CI (for example, /tag-and-rerun-ci)? I do not have CI-label permission.

Local validation: 142 related tests passed. The two focused responsiveness tests and the ASGI integration test fail with the expected starvation assertions on unmodified main; the ASGI test now queues real /v1/chat/completions and /ping requests before yielding to the event loop.

[anencore94]

@JustinTong0323 @merrymercy Gentle ping when you have a chance. This PR is ready for review and touches the OpenAI request-preprocessing path and TokenizerManager ownership boundary. The related unit and ASGI regression tests pass locally; I would especially appreciate feedback on the offloading and serialization approach. Thank you!

[anencore94]

Rebased this PR on the latest main; the previous merge conflict is resolved and the branch now includes the current MIN_BASE_SHA gate from #21065.

This is the one PR I would like to prioritize for review.

Impact:

• Synchronous OpenAI request conversion and regular tokenization can block the API/tokenizer event loop.
• That can delay lightweight liveness endpoints such as /ping and /health even while the backend is otherwise alive.
• The patch keeps the existing serialized preprocessing behavior, but runs the blocking work in a TokenizerManager-owned single-thread executor.

Local validation after rebase:

• ruff check on all changed files
• py_compile on all changed files
• scripts/ci/check_registered_tests.py
• git diff --check
• Focused CPU unit tests: test_serving_base_event_loop.py and test_tokenizer_manager_event_loop.py passed locally with a macOS-only torch.compile import stub used for collection.

The ASGI liveness test was not rerun successfully in my local macOS environment because collection now requires optional local dependencies such as mlx; it should be covered by CI.

Could an authorized maintainer trigger CI with /tag-and-rerun-ci and review the frontend offloading boundary? I would especially appreciate feedback from @JustinTong0323 on the OpenAI serving path, @slin1237 for Router/API ownership, and @merrymercy / @hnyls2002 for the TokenizerManager / manager boundary.

[anencore94]

Hi @merrymercy @hnyls2002 — gentle ping on this one. Routing to you two as the most active recent committers on tokenizer_manager.py / http_server.py (apologies if someone else is a better fit).

Quick summary for triage:

• Problem: synchronous request preprocessing blocks the TokenizerManager event loop, so /health stalls while tokens are still generating → Kubernetes liveness kills otherwise-healthy pods. We hit this in production at Toss Securities (271 gateway 502s from liveness restarts under load).
• Fix: offload the blocking preprocessing to a single-thread executor — no behavior change otherwise.
• Status: branch is up to date with main (0 behind), 142 tests pass locally (137 existing + 5 new, including an ASGI reproducer). The remaining red checks are the pr-test-*-finish gates — the underlying test stages show as skipped, so I believe CI just needs a maintainer trigger.

Happy to adjust scope or approach if you'd prefer a different design. Thanks!

[anencore94]

Rebased onto the latest main and resolved the conflicts.

@ch-wan @hnyls2002 — bumping this one. Request preprocessing on the tokenizer-manager path runs inline on the event loop, so a burst of large requests starves it: /health and the liveness path stop being served while the engine itself is perfectly fine. Under an orchestrator with a liveness probe, that turns a transient load spike into a restart of a healthy process. The PR moves the blocking part off the loop and adds CPU-only regression tests covering HTTP liveness during preprocessing.

On the red checks: all pr-gate / pr-test-*-finish, failing for a missing run-ci label rather than any real test failure — I'm not in .github/CI_PERMISSIONS.json so I can't trigger the lanes myself. Could a maintainer add run-ci? The three new tests are CPU-only and should be quick.

[yongqinwang-cmd]

We are seeing the same false-health-check issue in production with DeepSeek-V4-Flash on SGLang v0.5.16: synchronous request preprocessing can starve /health while inference is still making progress.

We backported this PR onto our existing v0.5.16 image and ran the same 448-concurrent-client load test with ~64k-token chat inputs. Before the patch, the baseline produced 4 new readiness timeouts in the first minute. With this PR applied, the same load ran for 10+ minutes with zero readiness timeouts, zero liveness timeouts, and zero request errors.

[anencore94]

@yongqinwang-cmd thank you — that's a much better validation than anything I could produce on my own, and the numbers line up exactly with what this patch is meant to fix. Independent reproduction on a different model and a different deployment, with a measured before/after (4 readiness timeouts in the first minute → zero over 10+ minutes at 448 concurrent clients with ~64k-token inputs), is the strongest evidence this PR has.

Worth stating plainly why the failure mode is worse than it looks: the health endpoint going quiet does not mean the engine is unhealthy. Inference keeps making progress the whole time — it is only the event loop that cannot get a turn to answer the probe. So an orchestrator with a liveness probe restarts a server that was working, drops every in-flight request on it, and the restart lands the load on the remaining replicas, which then starve the same way. A transient burst turns into a rolling outage, and nothing in the logs points at preprocessing.

@ch-wan @hnyls2002 @merrymercy — given a second party is now carrying this as a local patch in production, could we get this looked at? It is confined to the tokenizer-manager preprocessing path and ships three CPU-only regression tests.

The blocker is still purely mechanical: the red checks are pr-gate / pr-test-*-finish refusing to start because the PR has no run-ci label, so no test has ever actually run on it. I am not in .github/CI_PERMISSIONS.json, so /tag-run-ci-label is a no-op when I post it. Adding the label is all that is needed to get real signal here.

[anencore94]

Rebased onto current main and re-verified end to end. Putting the case for merging in one place, because I think this one has sat long enough.

The bug still reproduces on today's main

I re-ran the reproducer from #30770 against current main with only the new tests applied and no production change: both starvation tests fail. With this PR applied they pass. So this is not a historical report — the synchronous preprocessing path is still on the event loop today, two months after the issue was filed.

Concretely, OpenAIServingBase.handle_request() calls _convert_to_internal_request() synchronously (Jinja render + tokenizer.encode()), and TokenizerManager._tokenize_texts() has a synchronous fallback. Both run on the same uvicorn loop as /ping, /health and /health_generate.

Why the failure mode is worse than "a slow endpoint"

The engine is fine the whole time. Inference keeps making progress; only the loop cannot get a turn to answer the probe. So an orchestrator with a liveness probe restarts a healthy server, drops its in-flight requests, and pushes that load onto the remaining replicas — which then starve the same way. A transient burst becomes a rolling restart, and nothing in the logs points at preprocessing. That is why #30770 uses /ping rather than /health: it isolates loop responsiveness from generation health.

Independent production confirmation

@yongqinwang-cmd hit this on a different model and a different deployment, backported the patch, and measured it (see above): 4 readiness timeouts in the first minute before, zero over 10+ minutes after, at 448 concurrent clients with ~64k-token inputs. They are carrying it as a local patch today.

The change is small and deliberately conservative

+412/-11 across 5 files, of which 378 lines are tests. The production change is 29 added lines:

• A ThreadPoolExecutor(max_workers=1) — single worker on purpose. Preprocessing is moved off the loop but stays serialized, so this cannot trade event-loop starvation for CPU oversubscription under a container quota.
• copy_context().run(...) so contextvars survive the thread hop.
• Both blocking paths routed through it.

What I fixed in this push

Two things had rotted while the branch waited, both now handled:

• _tokenize_texts began consulting model_config.is_embedding_gemma after these tests were written, so the bare-manager test needed that attribute stubbed.
• The registered-test taxonomy (test/registered/<kind>/<subsystem>/) landed since the branch was opened, and the ASGI liveness test was outside it. Moved to test/registered/unit/entrypoints/; check_registered_tests.py and check_no_registered_tests_in_package.py now both pass.

Verified locally: the two CPU responsiveness tests pass with the patch and fail without it. The ASGI test could not run in my environment (mlx is not installable here), so I am relying on CI for that one.

What is actually blocking this

Per MAINTAINER.md, this PR never got past step 2 — no Merge Oncall was ever assigned — so step 3 never happened either and it has no run-ci label. Every red check above is pr-gate refusing to start; not one test has ever run on this branch in CI. I am not in .github/CI_PERMISSIONS.json, so /tag-and-rerun-ci is a no-op for me.

Step 4 permits pinging other area oncalls when the assigned one is unresponsive. This touches srt/entrypoints and srt/managers:

• Router / API — @slin1237
• Scheduler — @merrymercy @hnyls2002 @cctry

Could one of you add run-ci and take the review? If the executor approach is not the direction you want, I would genuinely rather hear that than keep the branch alive — but #30770 is a real bug on current main with a second party patching around it, so I do not think the right outcome is for it to keep aging.

[anencore94]

@hnyls2002 could you review the frontend preprocessing offload and authorize CI for 96a3df4? The current gate stops because run-ci is absent. The five focused CPU regressions pass, including the real ASGI chat/ping path, serialized preprocessing and context propagation. Full upstream execution is still pending.

This complements #30060: it removes synchronous HTTP/tokenizer preprocessing from the event loop, while that PR addresses scheduler health-signal IPC. The updated title check now passes. If another maintainer owns this path, a reviewer referral would help.

[anencore94]

@hnyls2002 following up on 96a3df4c2483: could you review the serialized frontend preprocessing offload and authorize the upstream CI run? The current gate still reports missing run-ci. The five focused CPU regressions, including the real chat/ping ASGI path, pass locally; upstream execution remains pending. The PR title check was corrected and now passes. A reviewer referral would also help if another owner should take this forward.

[anencore94]

@chromecast56 @JianWang97 Thanks for building on #30771 in #39716 and #44502 and preserving the attribution. I want to consolidate this work into one merge path; I am happy for either #30771 or #39716 to carry it, with the original implementation/tests and subsequent contributions credited.

I updated #30771 to main and removed the OpenAI conversion hunk already covered by #39771. I also revisited the shared-tokenizer review on #39716. With the actual regular-tokenization method and pinned tokenizers==0.23.2, a synthetic CPU WordLevel probe still makes loop-side padding wait behind the shared backend (~246–257 ms, versus ~0.1 ms with a separate backend). This is not a model-serving benchmark, and the current head does not yet address that ownership issue.

I am preparing a narrower revision that passes a worker-owned fast-tokenizer backend explicitly to regular tokenization, leaving unsupported tokenizer types on their existing synchronous path. This follows the isolation direction already developed in #39716 without taking over its broader endpoint/ordering work.

Would you prefer to fold the focused fix/tests into #39716, or use #30771 as its small prerequisite and fold #44502 back here? Jian, validation of the final candidate on your workload would be especially helpful. I will keep this PR open until we agree on the target and attribution, rather than closing or replacing anyone's work prematurely.

[anencore94]

@hnyls2002 @merrymercy I pushed e0a728540 with a smaller scope informed by the shared-tokenizer review on #39716: regular HF fast tokenization now uses a startup-cloned native backend, passed explicitly to its worker. Slow/custom tokenizers retain their existing path. The OpenAI conversion hunk already merged in #39771 is removed.

The 6 focused regressions pass (the unmodified base fails the 2 starvation cases); the related CPU suite passes 388 tests + 376 subtests. A synthetic probe with pinned tokenizers 0.23.2 also confirms that loop-side padding no longer waits behind this worker's backend. GPU short-prompt TTFT at low/high concurrency remains unvalidated.

Could you review this isolation boundary and authorize run-ci for the current head? The current gate stops at the missing label. I have also asked the authors of #39716/#44502 to consolidate into one merge path; I am happy to move this focused fix and its tests into #39716 with attribution if that is preferable.

[anencore94]

@hnyls2002 @merrymercy could you help choose the merge path for the remaining regular fast-tokenizer fix and authorize CI on e0a728540bb6?

#39771 already covers OpenAI conversion; this PR is limited to regular HF fast tokenization with a separate worker-owned backend. The recorded CPU suite passes 388 tests plus 376 subtests, including starvation regressions that fail on the unmodified base. Slow/custom tokenizers keep their synchronous behavior. GPU serving/short-prompt TTFT validation remains outstanding.

#39716 and #44502 are still open and overlap this work. Would you prefer this three-file fix as a prerequisite, or fold its implementation and regressions into one of those PRs with attribution preserved? I would like to avoid parallel merge paths. The current upstream tests are blocked on run-ci; I am not treating the local results as CI approval.