feat(tito): add DeepSeek V3.2 speculative session verification - #1628
Merged
Conversation
guapisolo
requested review from
Shi-Dong,
Zhichenzzz,
fzyzcjy,
jybsuper,
maocheng23,
yueming-yuan and
yushengsu-thu
as code owners
July 11, 2026 00:02
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for speculative decoding in sglang-based session verification, updates sglang parallel size argument handling to use shorter destination names (such as sglang_ep_size), and implements thinking mode forwarding for DeepSeek V3.2. The reviewer suggests avoiding hardcoded speculative decoding parameters in namespace_to_train_args and instead retrieving them dynamically from the namespace or model configuration.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Add a manual TP8/EP8 session test and expose the fixed EAGLE serving recipe through ModelConfig. Keep SGLang TP/DP/PP/EP namespace fields on Miles' short-name contract, and forward the effective V3.2 thinking mode so SGLang separates reasoning before TITO comparison.
Remove the old-to-new mapping table and select tp_size, dp_size, pp_size, or ep_size directly from SGLang's registered aliases without changing the CLI surface.
Document that argparse derives its destination from the first alias, which is why Miles selects SGLang's short parallel-size field names.
…lates/ Copy sglang's encoding_dsv32/encoding_dsv4 (0.5.14.dev37+gf8cfad3) into miles/utils/chat_template_utils/templates/ verbatim (provenance header only) and route the deepseek bridge plus the test reference encoders through the vendored modules. Miles-side render modifications land separately; the sglang protocol.Tool seam stays on sglang so tool schemas keep serializing with the server's defaults. The vendored files are excluded from pre-commit formatters so they stay byte-diffable against upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tool,user} surface
Upstream encoding_dsv32 gates every thinking block on last_user_idx, so a
new user turn strips historical think from the render and rewrites the
prefix — the reason V3.2 registered only the {tool} surface. Mirror
encoding_dsv4's gates in the vendored copy: drop_thinking=False now keeps
every assistant's thinking block and renders <think> openers position-
independently, while drop_thinking=True paths stay byte-identical to
upstream (corpus + raise parity tests). Pin drop_thinking=False on every
V3.2 surface and register {tool, user}; pure tool-loop histories render
byte-identically under either drop mode, so existing {tool} configs are
unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
guapisolo
force-pushed
the
fix/dsv32-thinking-parser-mode
branch
from
July 23, 2026 23:22
4f01de1 to
1e338c0
Compare
This was referenced Jul 23, 2026
guapisolo
added a commit
that referenced
this pull request
Jul 25, 2026
…server v2 The v5 multi-lineage design lands as an OPT-IN second implementation instead of replacing v1: --use-session-server grows an optional value (bare flag or "v1" = the untouched linear server, "v2" = tree serving), and the whole tree stack lives in miles/rollout/session/v2/. The v1 modules stay byte-identical to their pre-branch state. v2 package (carried from the v5 line, imports adjusted only): - session_state: always-branch serving over the forest — deepest attach point, suffix becomes the branch delta, non-extensions grow siblings or new roots; retry semantics moves entirely to the samples-op picker. Branch suffixes may carry client assistants (compaction); snapshot splice with zero-inheritance-root fallback on canonical-prefix divergence. Truncated-path extension is 409 (TruncatedGenerationError, defined here — v1 never raises it). --session-strict-append-only is the single-chain guard: branch shapes fail loud with attach diagnostics. - assembly: per-leaf fold (compute -> truncate -> fold), default pick (temporal-supersession retry trim; roots never trimmed; non-retry-shaped trees 422) and default merge (exactly-once completion masking over the surviving set, rewards keyed by response id, fold < agent < server metadata layering); both replaceable via --session-sample-picker-path / --session-merge-function-path (sync-only, loaded in-process). - core: v2 twin of SessionCore reusing v1's HTTP plumbing unchanged. Shared seams, each inert for v1 by construction: - sessions.py dispatches registry/core on the flag and forwards the collect body's "metadata" (agent semantic layer) only under v2. - codec: encode/decode parameterized by a fields tuple; v1 default keeps the wire byte-identical, v2 adds reward + per-sample metadata (COMPUTED_FIELDS_V2) with conditional overlay semantics on decode. - arguments: flag upgrade (nargs="?"), unknown values rejected, and the three v2 flags require --use-session-server v2. Tests: v2 HTTP matrix (tree pins incl. deep/root divergence branching, strict guard, truncation + compaction), session_state unit matrix, and the samples-op suite (golden, multi-leaf trim/masking/rewards, hook lanes) — all against a v2-flagged server; the restored v1 suites and the byte-exact A-list (git diff against the pre-branch base is empty) pin that the default path did not move. Rebased onto the tito/session PR stack (#1628/#1777/#1778 + #1759/#1760), with the stack's landed shape taken as authoritative: - supports_midpath_assistant_rerender (True on the base class) moves in here with its consumer, the v2 suffix gate. No family sets it False anymore: the vendored DeepSeek-V3.2 encoder renders position-independently under drop_thinking=False, so the gate text no longer names it; a stub family pins the fail-loud branch. - v2 serving fills SessionRecord.request_timestamp like v1, so folded samples keep their lifecycle req_ts across the wire. - codec: COMPUTED_FIELDS_V2 is re-expressed over the ValueSpec table (SAMPLES_VALUE_SPEC_V2 superset), same wire semantics. - v2 unit tests pin allowed_append_roles=["tool"] explicitly; the ctor default is tool+user since the append-roles PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Shi-Dong
reviewed
Jul 25, 2026
| @pytest.mark.parametrize("scenario", list(_UPSTREAM_PARITY_SCENARIOS), ids=list(_UPSTREAM_PARITY_SCENARIOS)) | ||
| @pytest.mark.parametrize("thinking", [False, True], ids=["chat", "thinking"]) | ||
| def test_drop_thinking_true_matches_upstream_sglang(scenario, thinking): | ||
| from sglang.srt.entrypoints.openai import encoding_dsv32 as upstream |
Contributor
There was a problem hiding this comment.
please hoist this import to the beginning of the script
Shi-Dong
reviewed
Jul 25, 2026
| def test_drop_thinking_true_raise_parity_with_upstream(): | ||
| # thinking mode + a post-last-user assistant without reasoning_content or | ||
| # tool_calls raises upstream; the vendored copy keeps that contract. | ||
| from sglang.srt.entrypoints.openai import encoding_dsv32 as upstream |
Contributor
There was a problem hiding this comment.
please hoist this import to the beginning of the script
Shi-Dong
reviewed
Jul 25, 2026
| # tool_calls raises upstream; the vendored copy keeps that contract. | ||
| from sglang.srt.entrypoints.openai import encoding_dsv32 as upstream | ||
|
|
||
| from miles.utils.chat_template_utils.templates import encoding_dsv32 as vendored |
Contributor
There was a problem hiding this comment.
please hoist this import to the beginning of the script
Keep the upstream and vendored encoder modules at test module scope so the parity helpers and exception checks share the reviewed dependencies without changing behavior.
Shi-Dong
reviewed
Jul 25, 2026
Contributor
There was a problem hiding this comment.
The PR description says that this file is vendored "verbatim" but it doesn't seem so. Perhaps you'd want to update the PR description.
Collaborator
Author
There was a problem hiding this comment.
no in this pr it's the same to keep consistancy behavior with existing code. template change happens in #1778
| reasoning_parser="deepseek-v3", | ||
| tool_call_parser="deepseekv32", | ||
| tito_model="deepseekv32", | ||
| allowed_append_roles=("tool",), |
Contributor
There was a problem hiding this comment.
Aren't we supposed to test ("tool", "user") here?
Keep the Miles-owned V3.2 encoder copy for drop_thinking while restoring V4 to SGLang encoding_dsv4. Narrow the pre-commit exclusion and parity-test import to the remaining V3.2 vendoring.
Shi-Dong
approved these changes
Jul 26, 2026
Shi-Dong
left a comment
Shi-Dong
left a comment
Contributor
There was a problem hiding this comment.
Overall LGTM. Added one comment and the PR description might need updates.
| num_gpus=8, | ||
| tp_size=8, | ||
| ep_size=8, | ||
| enable_spec=True, |
Contributor
There was a problem hiding this comment.
In the PR description it says that EAGLE fails with Dpsk v3.2? Why is speculative decoding enabled in this test?
Collaborator
Author
There was a problem hiding this comment.
Enable spec to speed up the test, it's rollout-only and can speed up a lot.
There is actually some issue for dsv32 + mtp. So I naively skip it.
guapisolo
added a commit
that referenced
this pull request
Jul 26, 2026
The session verifier and its e2e callers serialize through sglang's real argparse destinations (the renames landed upstream with #1628). Add a regression test pinning the parser contract: external CLI spellings parse into SGLang's ServerArgs short destinations (sglang_dp_size, sglang_pp_size, sglang_ep_size, sglang_attn_cp_size, with sglang_tp_size resolved by validation) so removed long destinations cannot silently come back. Co-authored-by: Jiajun Li <jiajun.li@radixark.ai> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeepSeek V3.2 TITO session verification plus a vendored encoder that keeps historical thinking.
Motivation
Add a manual TP8/EP8 session test and expose the fixed EAGLE serving recipe through ModelConfig. Keep SGLang TP/DP/PP/EP namespace fields on Miles' short-name contract, and forward the effective V3.2 thinking mode so SGLang separates reasoning before TITO comparison.
Upstream
encoding_dsv32gates every thinking block onlast_user_idx: a new user turn strips historical think and rewrites the rendered prefix, which is why V3.2 registered only the{tool}surface. Vendoring the encoder lets Miles fix this at the render level without forking sglang.Usage
The test launches one session verifier with
tp=8,ep=8, and EAGLE steps/top-k/draft-tokens set to2/1/3.Design Notes
encoding_dsv32is vendored verbatim undermiles/utils/chat_template_utils/templates/(provenance header only, formatter-excluded); V4 stays on sglang'sencoding_dsv4.protocol.Toolseam stays on sglang so tool schemas keep serializing with the server's defaults.drop_thinking=Falserenders every thinking block position-independently whiledrop_thinking=Truestays byte-identical to upstream.drop_thinking=False, which legalizes the{tool, user}surface.DeepSeekV32TITOTokenizerforwards the resolved thinking mode aschat_template_kwargs["thinking"];add_sglang_argumentsmaps parallel-size aliases to Miles' short Namespace fields.Verification
drop_thinking=Truepath to the installed sglang encoder.test_deepseek_v32_forwards_effective_thinking_mode,test_sglang_parallel_sizes_use_short_namespace_fields,test_namespace_to_train_args_enables_eagle_speculative_decoding.DeepseekV3ForCausalLMNextNlacksfuse_qkv_a_proj.Review Focus
encode_messagespass-through.DeepSeekV32TITOTokenizereffective-mode precedence.namespace_to_train_argsfixed EAGLE option serialization.