feat: minimax m3 dynamo changes

[indrajit96]

Overview:

Adds MiniMax M3 support to Dynamo’s OpenAI frontend path, including request-level thinking / thinking_mode handling, parser aliasing, special-token/EOS handling, prior tool-call message validation, and MiniMax-M3-VL model-card context-length detection for multimodal configs.

Details:

• Add MiniMax M3 thinking-mode support:
  - Normalize OpenAI thinking into both generic thinking and MiniMax-style thinking_mode.
  - Support enabled, disabled, and adaptive.
  - Make SGLang force-reasoning respect thinking_mode=disabled.

• Add MiniMax-M3-VL multimodal config support:

  • Read config.json.text_config.max_position_embeddings before falling back to tokenizer config.
  • This avoids using tokenizer sentinel/default context lengths for multimodal HF configs.

• Add MiniMax M3 reasoning behavior:

  • Detect MiniMax M3 prompt-injected <mm:think> instead of generic <think>.
  • Disable reasoning parsing when request args set thinking_mode=disabled.
  • Treat MiniMax M3 parser aliases as requiring special-token preservation.

• Add SGLang frontend MiniMax M3 handling:

  • Normalize Dynamo parser aliases like minimax_m3, minimax_m3_nom, and minimax-m3-nom to SGLang’s minimax-m3.
  • Make MiniMax M3 force-reasoning follow thinking_mode.
  • Strip trailing EOS token IDs before final decode so visible EOS text does not leak into content.

• Validate prior assistant tool-call messages:

  • Reject messages[*].tool_calls[*].function.arguments unless it is a valid JSON object string.
  • This catches bad tool-call history before chat-template rendering.

• Fix SGLang stream usage emission:

  • Clear pending usage after emitting it once.

Where should the reviewer start?

Start with:

• lib/llm/src/protocols/openai/chat_completions.rs
• lib/llm/src/preprocessor.rs
• components/src/dynamo/frontend/sglang_prepost.py

Then review:

• lib/llm/src/protocols/openai/validate.rs
• lib/llm/src/model_card.rs
• components/src/dynamo/frontend/tests/test_sglang_processor_unit.py

Related Issues

Related to ai-dynamo/frontend-crates#83

Summary by CodeRabbit

• New Features

  • Improved support for MiniMax M3 reasoning and tool-calling, including broader parser-name compatibility and better default reasoning behavior.
  • Added support for additional thinking modes: enabled, disabled, and adaptive.

• Bug Fixes

  • Streaming responses now strip trailing end-of-sequence tokens before sending partial output, reducing stray tokens in emitted text.
  • Improved model context-length detection for some multimodal models.
  • Added stricter validation for tool-call arguments to catch invalid JSON earlier.

Closes DIS-2270

[coderabbitai]

Walkthrough

The PR adds MiniMax-M3 reasoning and parser normalization across OpenAI, preprocessor, and Sglang frontend paths, validates assistant tool-call arguments as JSON objects, strips trailing EOS token IDs from streamed chunks, and adds a multimodal context-length fallback in model-card loading.

Changes

Reasoning and streaming handling

Layer / File(s)	Summary
OpenAI thinking mode and tool-call validation lib/llm/src/protocols/openai/chat_completions.rs, lib/llm/src/protocols/openai/validate.rs, lib/llm/src/http/service/openai.rs	thinking now normalizes into thinking_mode values, including adaptive, and assistant tool-call arguments are validated as JSON object strings before downstream rendering.
MiniMax-M3 reasoning gating lib/llm/src/preprocessor.rs	OpenAIPreprocessor now detects injected reasoning by parser, treats MiniMax M3 parser aliases as special-token-dependent, and disables reasoning when thinking_mode is disabled.
Sglang parser alias normalization components/src/dynamo/frontend/sglang_prepost.py, components/src/dynamo/frontend/tests/test_sglang_processor_unit.py	MiniMax-M3 parser aliases are normalized before parser creation, force-reasoning resolution, and guided decoding, with unit tests covering the alias mapping and related parser wiring.
EOS trimming in streaming output components/src/dynamo/frontend/sglang_prepost.py, components/src/dynamo/frontend/sglang_processor.py	SglangStreamingPostProcessor now receives EOS token IDs from both processor paths, strips trailing configured EOS tokens from finished chunks, and clears pending usage after usage emission.

Model card context fallback

Layer / File(s)	Summary
Multimodal context-length fallback lib/llm/src/model_card.rs	architectural_max_context_length now reads config.json.text_config.max_position_embeddings when the top-level config omits the value, with explicit errors for missing or invalid text_config data.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is mostly complete, but the required Related Issues section is incomplete and does not use the template's issue-link format.	Replace the raw PR link with either 'Closes #XXXX' or 'Confirmed — no related issue' and keep only one chosen path in the required section.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and clearly points to the MiniMax M3 Dynamo changes.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

lib/llm/src/protocols/openai/chat_completions.rs (1)

149-177: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve top-level request precedence when folding reasoning controls.

Line 149 and Line 170 use or_insert(...), but Line 177 still drops self.thinking. If a caller sends thinking={"type":"disabled"} or reasoning_effort alongside stale chat_template_args, the explicit request field is silently ignored. That is user-visible for MiniMax because OpenAIPreprocessor::is_reasoning_disabled_by_request() only consults chat_template_args, so reasoning can stay enabled even though the request disabled it. Overwrite these normalized keys or reject conflicting combinations before clearing the source field.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/llm/src/protocols/openai/chat_completions.rs` around lines 149 - 177,
Preserve request-field precedence in the OpenAI chat normalization logic: in the
`chat_completions` path that updates `chat_template_args`, the current
`or_insert` behavior lets stale template args win over explicit `thinking` or
`reasoning_effort` values before `self.thinking` is cleared. Update the
normalization in the relevant method so the request-provided values always
override existing `chat_template_args`, or detect and reject conflicting
combinations, and ensure
`OpenAIPreprocessor::is_reasoning_disabled_by_request()` still sees the intended
user request after `self.thinking` is set to `None`.

components/src/dynamo/frontend/sglang_processor.py (1)

581-605: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Clearing pending_usage at Line 583 drops the cached_tokens metric on the same flush.

pending_usage is set to None immediately after being attached to dynamo_out["usage"], but the metrics block at Line 603 reads pending_usage to derive cached_tokens. On the flush that emits usage (typically the finish chunk, where completion_usage arrives), _cached_tokens_from_usage(None) returns None, so cached_tokens is silently omitted from the metrics. Note Line 612 already resets pending_usage every flush, so the early clear is redundant for preventing cross-cycle reuse.

Capture the usage for metrics before clearing:

🐛 Proposed fix

                     envelope: dict[str, Any] = {"_dynamo_annotated": True}
+                    usage_for_metrics = pending_usage
                     if choice:
                         dynamo_out: dict[str, Any] = {
                             "id": request_id,
                             "choices": [choice],
                             "created": created_ts,
                             "model": request["model"],
                             "object": "chat.completion.chunk",
                         }
                         if pending_usage:
                             dynamo_out["usage"] = pending_usage
                             pending_usage = None
@@
                     cached_tokens = _cached_tokens_from_usage(pending_usage)
+                    cached_tokens = _cached_tokens_from_usage(usage_for_metrics)
                     if cached_tokens is not None:
                         metrics["cached_tokens"] = cached_tokens

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/sglang_processor.py` around lines 581 - 605,
The flush logic in sglang_processor is clearing pending_usage too early, which
causes the metrics block to miss cached_tokens on the same emission. In the
section that builds dynamo_out and metrics, preserve the usage value long enough
for _cached_tokens_from_usage to read it, then clear pending_usage only after
cached_tokens has been derived and added to metrics. Use the existing
pending_usage, dynamo_out["usage"], and _cached_tokens_from_usage symbols to
keep the cached token metric available on the finish chunk.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/frontend/sglang_processor.py`:
- Line 415: `SglangProcessor` is only forwarding a single EOS id, so models with
multiple EOS tokens won’t get correct stop handling or trailing-EOS stripping.
Update the request setup in `SglangProcessor` to use the full EOS id list
resolved by `eos_token_ids()` instead of `tokenizer.eos_token_id`, and pass that
same full list into `SglangStreamingPostProcessor` so both request construction
and streaming post-processing stay aligned.

In `@lib/llm/src/model_card.rs`:
- Around line 1445-1461: The fallback logic in
model_card::architectural_max_context_length is too broad because `.or_else(|_|
...)` treats any `config.json` failure as a missing field. Change the chain so
only “field not found” cases fall back to `text_config` or
`tokenizer_config.json`, while parse/type errors from `crate::file_json_field`,
the `text_config` lookup, or the `serde_json::from_value` step are surfaced
immediately. Keep the existing flow in `model_card.rs` but make the error
handling distinguish absent keys from malformed metadata so
`effective_context_length()` is not computed from a bad repo config.

---

Outside diff comments:
In `@components/src/dynamo/frontend/sglang_processor.py`:
- Around line 581-605: The flush logic in sglang_processor is clearing
pending_usage too early, which causes the metrics block to miss cached_tokens on
the same emission. In the section that builds dynamo_out and metrics, preserve
the usage value long enough for _cached_tokens_from_usage to read it, then clear
pending_usage only after cached_tokens has been derived and added to metrics.
Use the existing pending_usage, dynamo_out["usage"], and
_cached_tokens_from_usage symbols to keep the cached token metric available on
the finish chunk.

In `@lib/llm/src/protocols/openai/chat_completions.rs`:
- Around line 149-177: Preserve request-field precedence in the OpenAI chat
normalization logic: in the `chat_completions` path that updates
`chat_template_args`, the current `or_insert` behavior lets stale template args
win over explicit `thinking` or `reasoning_effort` values before `self.thinking`
is cleared. Update the normalization in the relevant method so the
request-provided values always override existing `chat_template_args`, or detect
and reject conflicting combinations, and ensure
`OpenAIPreprocessor::is_reasoning_disabled_by_request()` still sees the intended
user request after `self.thinking` is set to `None`.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c7adbc4e-cacd-4206-80f9-b8fea6d7eb84

📥 Commits

Reviewing files that changed from the base of the PR and between 15bdb11 and 67ecd53.

📒 Files selected for processing (8)

• components/src/dynamo/frontend/sglang_prepost.py
• components/src/dynamo/frontend/sglang_processor.py
• components/src/dynamo/frontend/tests/test_sglang_processor_unit.py
• lib/llm/src/http/service/openai.rs
• lib/llm/src/model_card.rs
• lib/llm/src/preprocessor.rs
• lib/llm/src/protocols/openai/chat_completions.rs
• lib/llm/src/protocols/openai/validate.rs

[KrishnanPrash]

Please take a look at the Failing M3 CI [Ref]:

FAILED [w86] components/src/dynamo/frontend/tests/test_sglang_processor_unit.py::test_minimax_m3_openai_disabled_thinking_sets_thinking_mode[request_update0] [27s/600s] - !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
FAILED [w87] components/src/dynamo/frontend/tests/test_sglang_processor_unit.py::test_minimax_m3_openai_disabled_thinking_sets_thinking_mode[request_update1] [27s/600s] - !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
FAILED [w88] components/src/dynamo/frontend/tests/test_sglang_processor_unit.py::test_minimax_m3_openai_disabled_thinking_sets_thinking_mode[request_update2] [26s/600s] - !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
FAILED [w89] components/src/dynamo/frontend/tests/test_sglang_processor_unit.py::test_minimax_m3_reasoning_effort_none_keeps_explicit_thinking_mode [27s/600s] - !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!