Skip to content

feat: sglang V4 (DeepSeek-V4) compatibility for decode handler #8678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davilu-nvidia wants to merge 1 commit into from
Closed

feat: sglang V4 (DeepSeek-V4) compatibility for decode handler #8678

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions components/src/dynamo/sglang/request_handlers/llm/decode_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,30 @@ def __init__(
generate_endpoint,
shutdown_event,
)
# H20 patch: set use_sglang_tokenizer from config
self.use_sglang_tokenizer = False # disagg mode
Comment on lines +69 to +70

@coderabbitai coderabbitai Bot Apr 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don't hard-code use_sglang_tokenizer after super().__init__().

BaseWorkerHandler.__init__() already used the config value to build request-parsing state, so forcing this field to False here makes the handler internally inconsistent and bypasses the new flag for aggregated serving. If decode mode truly needs a fixed tokenizer setting, apply it before super().__init__() or rebuild the dependent state afterward.

🛠️ Suggested fix 
-        # H20 patch: set use_sglang_tokenizer from config
-        self.use_sglang_tokenizer = False  # disagg mode
+        # Preserve the configured tokenizer mode; the base class already
+        # initialized its dependent state from this value.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# H20 patch: set use_sglang_tokenizer from config
self.use_sglang_tokenizer = False # disagg mode
# Preserve the configured tokenizer mode; the base class already
# initialized its dependent state from this value.
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/src/dynamo/sglang/request_handlers/llm/decode_handler.py` around
lines 69 - 70, The handler currently overwrites self.use_sglang_tokenizer =
False after calling BaseWorkerHandler.__init__(), which breaks the config-driven
request-parsing state; update DecodeHandler so the tokenizer flag is set
consistently by either assigning use_sglang_tokenizer from the config before
calling BaseWorkerHandler.__init__() or, if it must be forced, rebuild any
dependent state (the request-parsing state initialized in
BaseWorkerHandler.__init__) immediately after setting it; reference the
DecodeHandler.__init__ method and BaseWorkerHandler.__init__ and ensure no
post-init hard-coded assignment leaves the handler inconsistent.

if self.serving_mode == DisaggregationMode.DECODE:
logging.info(
"Decode worker handler initialized (disaggregated decode mode)"
)
else:
logging.info("Decode worker handler initialized (aggregated mode)")

@staticmethod
def _get_guided_decoding_params(guided_decoding=None):
"""Stub for guided decoding - not needed for E2E test."""
return {}

@staticmethod
def _resolve_lora(request):
"""Stub for LoRA resolution."""
return {}

@staticmethod
def _session_kwargs(request):
"""Stub for session kwargs."""
return {}
Comment on lines +78 to +91

@coderabbitai coderabbitai Bot Apr 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n components/src/dynamo/sglang/request_handlers/llm/decode_handler.py | head -120

Repository: ai-dynamo/dynamo

Length of output: 4903

🏁 Script executed:

# Check if there's a parent class definition in the file
rg "class.*\(" components/src/dynamo/sglang/request_handlers/llm/decode_handler.py -A 3

Repository: ai-dynamo/dynamo

Length of output: 214

🏁 Script executed:

# Look for imports to understand inheritance
rg "^from|^import" components/src/dynamo/sglang/request_handlers/llm/decode_handler.py | head -20

Repository: ai-dynamo/dynamo

Length of output: 589

🏁 Script executed:

# Find and examine the parent class BaseWorkerHandler
find . -name "handler_base.py" -path "*/sglang/*" | head -5

Repository: ai-dynamo/dynamo

Length of output: 124

🏁 Script executed:

# Search for the parent class and its method implementations
rg "def _get_guided_decoding_params|def _resolve_lora|def _session_kwargs" --type py -B 2 -A 5

Repository: ai-dynamo/dynamo

Length of output: 4948

🏁 Script executed:

# Search for calls to these stub methods to understand expected contract
rg "_get_guided_decoding_params|_resolve_lora|_session_kwargs" --type py -B 2 -A 2

Repository: ai-dynamo/dynamo

Length of output: 8601

🏁 Script executed:

# Get more context from the base class implementation
sed -n '1,250p' components/src/dynamo/sglang/request_handlers/handler_base.py | grep -A 20 "_resolve_lora"

Repository: ai-dynamo/dynamo

Length of output: 42

🏁 Script executed:

# Check the full implementations in base class
sed -n '1,400p' components/src/dynamo/sglang/request_handlers/handler_base.py | grep -A 15 "def _session_kwargs"

Repository: ai-dynamo/dynamo

Length of output: 42

🏁 Script executed:

# Check the full implementation of _get_guided_decoding_params in base
sed -n '1,400p' components/src/dynamo/sglang/request_handlers/handler_base.py | grep -A 15 "def _get_guided_decoding_params"

Repository: ai-dynamo/dynamo

Length of output: 42

🏁 Script executed:

# Get the size and read the handler_base.py file
wc -l components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 127

🏁 Script executed:

# Read the handler_base.py file to find the method implementations
cat components/src/dynamo/sglang/request_handlers/handler_base.py | head -500

Repository: ai-dynamo/dynamo

Length of output: 18776

🏁 Script executed:

# Search for the implementations of these methods in handler_base.py
grep -n "def _resolve_lora\|def _session_kwargs\|def _get_guided_decoding_params" components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 256

🏁 Script executed:

# Get the implementation of _resolve_lora method starting from the line
sed -n '780,810p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 1122

🏁 Script executed:

# Get the implementation of _session_kwargs method
sed -n '810,850p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 1647

🏁 Script executed:

# Get the implementation of _get_guided_decoding_params method
sed -n '850,900p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 2241

🏁 Script executed:

# Get the actual implementations starting from the correct line numbers
sed -n '302,320p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 759

🏁 Script executed:

# Get _session_kwargs implementation
sed -n '1044,1070p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 1334

🏁 Script executed:

# Get _get_guided_decoding_params implementation
sed -n '1057,1080p' components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 1036

🏁 Script executed:

# Verify the calls to these methods in DecodeWorkerHandler and check if they're used correctly
sed -n '78,300p' components/src/dynamo/sglang/request_handlers/llm/decode_handler.py | grep -A 5 "_resolve_lora\|_session_kwargs\|_get_guided_decoding_params"

Repository: ai-dynamo/dynamo

Length of output: 1061

🏁 Script executed:

# Check return type annotations and what lora_path is used for
grep -n "lora_path" components/src/dynamo/sglang/request_handlers/llm/decode_handler.py | head -20

Repository: ai-dynamo/dynamo

Length of output: 312

🏁 Script executed:

# Check if DecodeWorkerHandler uses LoraMixin
grep -n "class DecodeWorkerHandler\|LoraMixin\|_init_lora_tracking" components/src/dynamo/sglang/request_handlers/llm/decode_handler.py

Repository: ai-dynamo/dynamo

Length of output: 109

🏁 Script executed:

# Check BaseWorkerHandler class definition and its mixins
sed -n '1100,1150p' components/src/dynamo/sglang/request_handlers/handler_base.py | grep -A 20 "class BaseWorkerHandler"

Repository: ai-dynamo/dynamo

Length of output: 42

🏁 Script executed:

# Search for BaseWorkerHandler class definition in the full file
grep -n "^class BaseWorkerHandler" components/src/dynamo/sglang/request_handlers/handler_base.py

Repository: ai-dynamo/dynamo

Length of output: 153

Remove these method stubs—they return incorrect types and break LoRA resolution.

The stubs override working parent methods from LoraMixin and BaseGenerativeHandler. Specifically:

  • _resolve_lora() returns {} instead of Optional[str], causing if lora_path: to evaluate as truthy even for empty dicts; it also cannot access self.lora_id_for_name because it is @staticmethod.
  • _session_kwargs() and _get_guided_decoding_params() drop routing and guided decoding logic.

Delete these three methods to inherit the parent implementations.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/src/dynamo/sglang/request_handlers/llm/decode_handler.py` around
lines 78 - 91, The three static method stubs _get_guided_decoding_params,
_resolve_lora, and _session_kwargs in decode_handler.py override parent
implementations and return incorrect types/lose access to instance methods
(e.g., self.lora_id_for_name) causing LoRA resolution and guided decoding to
break; remove these three method definitions so the class will inherit the
correct implementations from LoraMixin and BaseGenerativeHandler, restoring
proper Optional[str] return behavior for _resolve_lora and the routing/guided
decoding logic for _session_kwargs and _get_guided_decoding_params.


def cleanup(self) -> None:
"""Shutdown the engine and cleanup resources."""
super().cleanup()
Expand Down Expand Up @@ -308,7 +325,7 @@ async def generate(
**input_param,
sampling_params=sampling_params,
stream=True,
return_routed_experts=return_routed_experts,
# return_routed_experts=return_routed_experts,
bootstrap_host=bootstrap_info["bootstrap_host"],
bootstrap_port=bootstrap_info["bootstrap_port"],
bootstrap_room=bootstrap_info["bootstrap_room"],
Expand Down Expand Up @@ -346,7 +363,7 @@ async def generate(
video_data=video_data,
sampling_params=sampling_params,
stream=True,
return_routed_experts=return_routed_experts,
# return_routed_experts=return_routed_experts,
external_trace_header=trace_header,
rid=trace_id,
data_parallel_rank=dp_rank,
Expand Down
Loading