feat(sa-bench): add sglang DeepSeek-V4 tokenizer (depends on #71)

[YAMY1234]

Summary

Extends #71's deepseek_v4 custom_tokenizer option with a sglang backend path, so sa-bench can accurately tokenize DeepSeek-V4-Pro prompts for sglang servers the same way it already does for vllm servers in #71.

Why this is needed. DeepSeek-V4 ships no Hugging Face chat template, so tokenizer.apply_chat_template() raises ValueError. Sglang's server solves this internally by replacing the HF template path with a hard-coded DSML encoder (encoding_dsv4.encode_messages) whenever arch == "DeepseekV4ForCausalLM" — see sgl-project/sglang#23600. Without a matching client-side encoder, the sa-bench tokenizer silently falls back to raw-text tokenization, so input_tokens reported by the client no longer matches the server's #new-token. This in turn skews ISL, TPOT, TTFT, and MTP accept-rate accounting.

What we add. There is no client-side DeepseekV4Tokenizer package on the sglang side (vllm has vllm.tokenizers.deepseek_v4; sglang has none). So we vendor sglang's own server-side encoder (encoding_dsv4.py from sgl-project/sglang PR #23600, commit f5d03db853862c8fb0e805df591bed883a71868b) under sa-bench/tokenizers/ and wrap it in a HF-compatible SGLangDeepseekV4Tokenizer.apply_chat_template(). The wrapper mirrors exactly what sglang server does:

1. Insert an empty system message if missing (matches serving_chat._resolve_chat_encoding_spec).
2. Default thinking_mode="chat", reasoning_effort=None (matches sglang defaults).
3. Call vendored encode_messages(...) to render the raw DSML string.
4. hf_tokenizer.encode(..., add_special_tokens=False) (the encoder already adds <｜begin▁of▁sentence｜>).

The vendored file carries Apache-2.0 header + upstream commit SHA so it can be dropped when sglang publishes an official client-side package.

Routing is backend-aware — no implicit fallback. Per reviewer preference, we do not fall back to vllm if sglang install is present or vice versa; ambiguity is rejected:

if backend == "sglang":
    return SGLangDeepseekV4Tokenizer.from_pretrained(...)
if backend in (None, "vllm"):
    return vllm.tokenizers.deepseek_v4.DeepseekV4Tokenizer.from_pretrained(...)
raise ValueError(f"custom_tokenizer='deepseek_v4' does not support backend={backend!r}")

benchmark_serving.py passes args.backend into get_tokenizer(...).

Files

• src/srtctl/benchmarks/scripts/sa-bench/tokenizers/__init__.py — new package
• src/srtctl/benchmarks/scripts/sa-bench/tokenizers/_sglang_encoding_dsv4.py — vendored (Apache-2.0), 840 lines, unmodified upstream content
• src/srtctl/benchmarks/scripts/sa-bench/tokenizers/sglang_deepseek_v4.py — HF-compatible wrapper
• backend_request_func.py — adds backend kwarg to get_tokenizer, splits deepseek_v4 into explicit sglang / vllm branches
• benchmark_serving.py — plumbs args.backend through

Relationship to other PRs

• Depends on Combine DeepSeek V4 recipe and tokenizer changes #71 (aflowers/dsv4-pr67-pr68) — that PR introduces the deepseek_v4 custom_tokenizer option for vllm; this PR extends it for sglang.
• Orthogonal to feat: DeepSeek-V4-Pro perf recipes for GB300 / GB200 (1k/1k agg) #70 — recipes for DeepSeek-V4-Pro (including the gb300 1-node agg MTP recipe used for smoke testing) are added by elvischenv in feat: DeepSeek-V4-Pro perf recipes for GB300 / GB200 (1k/1k agg) #70. This PR intentionally adds no recipe files.

Test plan

• python -m py_compile passes on all three new / modified files.
• Module import: from tokenizers.sglang_deepseek_v4 import SGLangDeepseekV4Tokenizer succeeds with the vendored encoder present.
• Offline equivalence check: SGLangDeepseekV4Tokenizer(...).apply_chat_template(messages) token IDs match server-side tokenizer.encode(encode_messages(messages)) for a representative GSM8K prompt.
• Online smoke: short sa-bench run against recipes/gb300-fp4/1k1k-dsv4/agg-low-latency.yaml (from feat: DeepSeek-V4-Pro perf recipes for GB300 / GB200 (1k/1k agg) #70) with this tokenizer, asserting client input_tokens == server #new-token.

[YAMY1234]

Superseded by a clean PR based on main — see follow-up.