feat(sglang): support --frontend-decoding for aggregated multimodal

[furionw]

Why

The dynamo.sglang backend ignored --frontend-decoding: every multimodal
request went through SGLang's URL → HTTP → PIL decode path in the worker's
P0 thread. The dynamo.vllm backend has handled this for a while — the
Rust frontend already decodes images and ships pixels via NIXL RDMA when
the backend advertises a MediaDecoder at registration. This PR teaches
the SGLang backend to advertise the same and consume Decoded items via
the shared ImageLoader, removing the GIL-bound decode hop on the worker
side.

What Change

• New --frontend-decoding / DYN_SGL_FRONTEND_DECODING flag mirroring vLLM.
• Backend registers MediaDecoder + MediaFetcher with the Rust frontend.
• DecodeWorkerHandler aggregated arm: load Decoded items via ImageLoader, hand PIL list to sgl.Engine.async_generate(image_data=...).
• Validation rejects FD + --multimodal-{encode-worker,worker} (EPD needs URLs for MMEncoder).

Test Plan

• CPU unit tests cover arg validation + Decoded → PIL → engine wiring.
• H100 smoke (Qwen3-VL-2B + COCO image) via new examples/backends/sglang/launch/agg_vision_fd.sh returns a coherent caption.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Introduced --frontend-decoding configuration option across SGLang, TRT-LLM, and vLLM backends for frontend-based image decoding in multimodal models with NIXL RDMA support.
  • Updated example launch scripts with frontend decoding capability and configurable KV-cache page size.

• Tests

  • Added comprehensive test coverage validating frontend decoding functionality, configuration constraints, and multimodal image handling.

[coderabbitai]

Walkthrough

This PR implements frontend decoding support for multimodal inference across multiple backends. It introduces a shared configuration module, integrates the feature into SGLang with conditional image loading, refactors TRT-LLM and vLLM to use shared configuration helpers, and adds comprehensive tests and example scripts.

Changes

Frontend Decoding Multi-Backend Integration

Layer / File(s)	Summary
Shared frontend decoding configuration module components/src/dynamo/common/configuration/groups/frontend_decoding_args.py	Introduces add_frontend_decoding_arg helper that registers the --frontend-decoding negatable boolean flag with environment variable DYN_{env_prefix}_FRONTEND_DECODING across all backends.
SGLang backend configuration, validation, and wiring components/src/dynamo/sglang/backend_args.py	SGLang imports and wires the shared helper, adds frontend_decoding config field, and validates incompatibility with EPD multimodal worker topologies.
SGLang DecodeWorkerHandler frontend decoding implementation components/src/dynamo/sglang/request_handlers/llm/decode_handler.py	Handler gains enable_frontend_decoding parameter and implements dual-path aggregated-mode image handling: frontend-decoded path loads images via ImageLoader.load_image_batch to PIL objects; legacy path extracts media URLs as strings.
SGLang initialization and model registration wiring components/src/dynamo/sglang/init_llm.py, components/src/dynamo/sglang/register.py	init_decode passes flag to handler; _register_model_with_runtime_config conditionally constructs and passes media_decoder/media_fetcher objects during model registration.
TRT-LLM shared flag refactoring and optional import removal components/src/dynamo/trtllm/backend_args.py, components/src/dynamo/trtllm/workers/llm_worker.py	TRT-LLM backend replaces inline flag definition with shared helper and removes optional-import pattern by unconditionally requiring MediaDecoder and MediaFetcher.
vLLM shared flag refactoring and optional import removal components/src/dynamo/vllm/backend_args.py, components/src/dynamo/vllm/main.py	vLLM backend replaces inline flag definition with shared helper and removes optional-import pattern by unconditionally requiring MediaDecoder and MediaFetcher.
SGLang frontend decoding unit tests components/src/dynamo/sglang/tests/test_sglang_frontend_decoding.py	Tests validate configuration rules, handler initialization, and aggregated-mode image handling paths including frontend-decoded PIL images, URL strings, and no-images cases.
Example script and E2E integration tests examples/backends/sglang/launch/agg_vision.sh, tests/serve/test_sglang.py	Script adds --frontend-decoding flag and NIXL library discovery; E2E test configuration multimodal_agg_fd_qwen exercises the feature with base64 image payloads.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding frontend-decoding support to the sglang backend for aggregated multimodal requests.
Description check	✅ Passed	The description covers Why, What Change, and Test Plan sections with sufficient detail about the feature implementation and testing approach, though it lacks explicit reference to related GitHub issues.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (3)

components/src/dynamo/sglang/register.py (2)

42-44: ⚡ Quick win

Move DYN_MM_ALLOW_INTERNAL to argparse-backed config in SGLang.

This new env read introduces another runtime tunable outside DynamoSGLangArgGroup / DynamoSGLangConfig. Please thread it through config instead of reading env directly here.

Based on learnings, "In the SGLang Dynamo wrapper (components/src/dynamo/sglang), replace inline environment reads (os.environ.get()) of tunables with argparse-driven configuration."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/sglang/register.py` around lines 42 - 44, Replace the
direct os.getenv read of DYN_MM_ALLOW_INTERNAL in register.py by adding a
boolean option to the existing argparse/config flow: add a field to
DynamoSGLangArgGroup (e.g. --dyn-mm-allow-internal) and expose it on
DynamoSGLangConfig, have the code that constructs/receives the config pass
config.dyn_mm_allow_internal into register.py instead of reading the env, and
then call media_fetcher.allow_direct_ip(config.dyn_mm_allow_internal) and
media_fetcher.allow_direct_port(config.dyn_mm_allow_internal); remove the
os.getenv usage and the local allow_internal variable and ensure the new arg has
a default matching the previous behavior.

---

85-86: ⚡ Quick win

Use direct config field access instead of getattr for frontend_decoding.

dynamo_args is a known typed config object where frontend_decoding: bool = False is explicitly defined; direct attribute access is safer and keeps schema changes visible.

♻️ Suggested local fix

-    if getattr(dynamo_args, "frontend_decoding", False):
+    if dynamo_args.frontend_decoding:

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/sglang/register.py` around lines 85 - 86, Replace the
getattr check with direct attribute access on the typed config object: use
dynamo_args.frontend_decoding instead of getattr(dynamo_args,
"frontend_decoding", False). Update the conditional that calls
_build_media_decoder_and_fetcher() to read dynamo_args.frontend_decoding so the
schema is explicit and type-safe (this surfaces type-checking and schema changes
via dynamo_args.frontend_decoding).

components/src/dynamo/sglang/request_handlers/llm/decode_handler.py (1)

515-515: 💤 Low value

Prefer explicit check over assert for invariant.

The Python guideline recommends avoiding assert for runtime validation in favor of explicit if ...: raise ..., even for invariants. This provides a clearer error message and is not disabled by -O optimization.

♻️ Suggested refactor

-                assert self._image_loader is not None
+                if self._image_loader is None:
+                    raise RuntimeError(
+                        "ImageLoader is None despite frontend_decoding=True; "
+                        "this is a programming error in DecodeWorkerHandler.__init__"
+                    )

As per coding guidelines: "Do not use assert for runtime validation (especially for config/request validation); use explicit if ...: raise ...."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/sglang/request_handlers/llm/decode_handler.py` at line
515, Replace the runtime assert with an explicit check: in the class/method
where the line "assert self._image_loader is not None" appears (e.g.,
DecodeHandler or its decode method), change it to an if statement that raises a
clear exception (ValueError or RuntimeError) with a helpful message like "image
loader not initialized" when self._image_loader is None; this ensures the
invariant is enforced at runtime and not skipped under -O optimization.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@components/src/dynamo/sglang/register.py`:
- Around line 42-44: Replace the direct os.getenv read of DYN_MM_ALLOW_INTERNAL
in register.py by adding a boolean option to the existing argparse/config flow:
add a field to DynamoSGLangArgGroup (e.g. --dyn-mm-allow-internal) and expose it
on DynamoSGLangConfig, have the code that constructs/receives the config pass
config.dyn_mm_allow_internal into register.py instead of reading the env, and
then call media_fetcher.allow_direct_ip(config.dyn_mm_allow_internal) and
media_fetcher.allow_direct_port(config.dyn_mm_allow_internal); remove the
os.getenv usage and the local allow_internal variable and ensure the new arg has
a default matching the previous behavior.
- Around line 85-86: Replace the getattr check with direct attribute access on
the typed config object: use dynamo_args.frontend_decoding instead of
getattr(dynamo_args, "frontend_decoding", False). Update the conditional that
calls _build_media_decoder_and_fetcher() to read dynamo_args.frontend_decoding
so the schema is explicit and type-safe (this surfaces type-checking and schema
changes via dynamo_args.frontend_decoding).

In `@components/src/dynamo/sglang/request_handlers/llm/decode_handler.py`:
- Line 515: Replace the runtime assert with an explicit check: in the
class/method where the line "assert self._image_loader is not None" appears
(e.g., DecodeHandler or its decode method), change it to an if statement that
raises a clear exception (ValueError or RuntimeError) with a helpful message
like "image loader not initialized" when self._image_loader is None; this
ensures the invariant is enforced at runtime and not skipped under -O
optimization.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4af8733b-1478-4342-b801-412894e9cfd6

📥 Commits

Reviewing files that changed from the base of the PR and between 80e67da and 7b63048.

📒 Files selected for processing (12)

• components/src/dynamo/common/configuration/groups/frontend_decoding_args.py
• components/src/dynamo/sglang/backend_args.py
• components/src/dynamo/sglang/init_llm.py
• components/src/dynamo/sglang/register.py
• components/src/dynamo/sglang/request_handlers/llm/decode_handler.py
• components/src/dynamo/sglang/tests/test_sglang_frontend_decoding.py
• components/src/dynamo/trtllm/backend_args.py
• components/src/dynamo/trtllm/workers/llm_worker.py
• components/src/dynamo/vllm/backend_args.py
• components/src/dynamo/vllm/main.py
• examples/backends/sglang/launch/agg_vision.sh
• tests/serve/test_sglang.py

[furionw]

/ok to test 39cd494