Skip to content

Honor direct chat no-think requests - #647

Merged
i386 merged 2 commits into
Mesh-LLM:mainfrom
IvGolovach:codex/direct-model-reasoning-conformance
May 24, 2026
Merged

Honor direct chat no-think requests#647
i386 merged 2 commits into
Mesh-LLM:mainfrom
IvGolovach:codex/direct-model-reasoning-conformance

Conversation

@IvGolovach

@IvGolovach IvGolovach commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Direct OpenAI-compatible chat requests now preserve explicit no-thinking intent when they are forwarded to downstream model backends.

This canonicalizes caller intent such as reasoning_effort: "none", reasoning.enabled=false, provider thinking aliases, and thinking_budget=0 into chat_template_kwargs.enable_thinking=false before forwarding. Existing chat_template_kwargs values remain authoritative and are not overwritten.

Why

Issue #636 showed that direct pinned-model requests could still surface <think> content even when the caller explicitly requested no thinking.

The typed Skippy path already understands these options, but the host proxy forwarding path could pass the original request body through without translating the no-thinking intent into the backend-compatible chat-template shape. That meant behavior depended on every downstream backend understanding every caller-specific reasoning knob.

What changed

  • Canonicalizes /v1/chat/completions and translated /v1/responses bodies into chat_template_kwargs.enable_thinking when explicit thinking/no-thinking intent is present.
  • Extends host-runtime forwarding detection so reasoning/template knobs trigger JSON body normalization before proxying.
  • Preserves existing chat_template_kwargs overrides and unrelated request fields.
  • Preserves nested no-thinking precedence, so reasoning.enabled=false wins over reasoning.effort="low" before forwarding.
  • Adds regressions for both the normalization layer and the relayed host-runtime forwarding path.
  • Adds Skippy-server coverage for the top-level reasoning_effort: "none" request shape.

Scope relative to #645

This PR stays scoped to request-side compatibility normalization before the direct forwarding/proxy path sends a body to a downstream OpenAI-compatible backend.

#645 is the broader guardrail/output-contract layer and should continue to own tool rescue, structured-output retries, and output cleanup. This PR does not try to duplicate that middleware path, and it should remain useful even while #645 settles because it does not depend on opt-in guardrails.

Compatibility

This is an HTTP-body-only compatibility fix.

No protobuf, gossip, mesh protocol, catalog, or Skippy ABI changes. Older peers and existing model-serving paths should continue to ignore unknown request fields as before.

Branch integrity

  • Base branch: main
  • Validated base: origin/main@c7948d7181f26ed8cf99f9389a7deb8988162a32
  • Branch state: 0 behind / 2 ahead
  • Head commit: a915c43199bbee9046c1e412ef2e24cf5f1ddeba

Validation

  • Validation tier: Tier 2R - narrow post-review correction on top of the existing OpenAI-compatible direct chat request normalization PR.
  • git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at c7948d7181f26ed8cf99f9389a7deb8988162a32.
  • git diff --check: PASS, no output
  • git diff --cached --check: PASS, no output
  • cargo fmt --all: PASS
  • cargo fmt --all -- --check: PASS
  • cargo test -p openai-frontend normalize_chat_reasoning_enabled_false_wins_over_nested_effort --lib: FAIL before fix, reproduced reviewer edge case.
  • cargo test -p openai-frontend normalize_chat --lib: PASS, 4 passed
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_reasoning_enabled_false_wins_over_nested_effort_before_forwarding --lib: PASS, 1 passed
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_reasoning_effort_none_is_canonicalized_before_forwarding --lib: PASS, 1 passed
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_existing_template_kwargs_survive_forwarding_rewrite --lib: PASS, 1 passed
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p skippy-server top_level_reasoning_effort_none_turns_off_chat_template_thinking --lib: PASS, 1 passed
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS
  • LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p openai-frontend -p skippy-server -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS
  • Ledger: not applicable - not required for selected validation tier/change family.
  • Version: not applicable - post-review compatibility precedence correction only; no release/version sync required.
  • Remote CI: PASS on final SHA a915c43199bbee9046c1e412ef2e24cf5f1ddeba (PR Builds, PR Docker Build, and PR Quality Checks).
  • Not run: live MiniMax/Qwen direct-model smoke - not required for this deterministic precedence correction.
  • Not run: two-node mesh or agent harness - not required for selected Tier 2R correction.

Rollback

Revert this PR.

DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.

Residual risk

The original hosted-model symptom should still be confirmed against the affected live endpoint. This PR proves the deterministic request-normalization and forwarding path locally, but does not claim to solve unrelated <think> leakage outside that path.

Validation
* Validation tier: Tier 2 - narrow OpenAI-compatible direct chat request normalization and host forwarding conformance for issue Mesh-LLM#636.
* git fetch --no-tags origin main:refs/remotes/origin/main: PASS, updated origin/main to c7948d7.
* git rebase origin/main: PASS.
* git diff --check origin/main...HEAD: PASS, no output
* git diff --check: PASS, no output
* git diff --cached --check: PASS, no output
* cargo fmt --all -- --check: PASS
* cargo test -p openai-frontend normalize_chat --lib: PASS, 3 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_reasoning_effort_none_is_canonicalized_before_forwarding --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_existing_template_kwargs_survive_forwarding_rewrite --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p skippy-server top_level_reasoning_effort_none_turns_off_chat_template_thinking --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p openai-frontend -p skippy-server -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - OpenAI-compatible request normalization only; no release/version sync required.
* Not run: live MiniMax/Qwen direct-model smoke - no local GGUF/runtime endpoint was available; deterministic forwarding and skippy template-option coverage prove the code path.

Rollback
* git revert HEAD
@IvGolovach
IvGolovach requested a review from ndizazzo May 23, 2026 01:38
@ndizazzo

ndizazzo commented May 23, 2026

Copy link
Copy Markdown
Collaborator

This might conflict with #645 - there's some work in there to correct think tags via the middleware layered approach, which I think is going to be more "canonical" for tweaking model output from backends to shape it

If that PR doesn't already entirely fix the issue, we might consider layering select changes from this one into the surfaces it adds instead

@i386

i386 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

One thing I noticed: reasoning.enabled: false can get flipped back on if the same request also has something like reasoning.effort: "low". That seems different from the existing normalize_reasoning_template_options behavior, where enabled: false wins and keeps thinking off. Could you make this forwarding normalizer follow the same precedence so direct-model requests do not accidentally re-enable thinking?

Validation
* Validation tier: Tier 2R - narrow post-review correction for direct chat reasoning precedence in OpenAI-compatible request forwarding.
* git diff --check: PASS, no output
* git diff --cached --check: PASS, no output
* cargo fmt --all: PASS
* cargo fmt --all -- --check: PASS
* cargo test -p openai-frontend normalize_chat_reasoning_enabled_false_wins_over_nested_effort --lib: FAIL before fix, reproduced reviewer edge case.
* cargo test -p openai-frontend normalize_chat --lib: PASS, 4 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_reasoning_enabled_false_wins_over_nested_effort_before_forwarding --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_reasoning_effort_none_is_canonicalized_before_forwarding --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime chat_existing_template_kwargs_survive_forwarding_rewrite --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p skippy-server top_level_reasoning_effort_none_turns_off_chat_template_thinking --lib: PASS, 1 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p openai-frontend -p skippy-server -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - post-review compatibility precedence correction only; no release/version sync required.
* Not run: live MiniMax/Qwen direct-model smoke - not required for this deterministic precedence correction.
* Not run: two-node mesh or agent harness - not required for selected Tier 2R correction.

Rollback
* git revert HEAD
@IvGolovach

Copy link
Copy Markdown
Collaborator Author

Thanks both, I looked through this carefully and updated the PR.

For the precedence edge case: reasoning.enabled=false now wins over nested reasoning.effort="low" before forwarding, matching the typed Skippy behavior. I added regressions at both the OpenAI normalization layer and the host-runtime forwarding path so this does not drift again.

For the #645 overlap: I kept this PR scoped to request-side compatibility normalization for the direct forwarding/proxy path. #645 still feels like the right home for the broader guardrail/output-contract layer, but this fix covers the raw forwarded request shape before any guardrail middleware is involved. The PR body is updated with the scope and validation, and CI is green on the final SHA.

@i386

i386 commented May 24, 2026

Copy link
Copy Markdown
Collaborator

LGTM

@i386
i386 merged commit 4f6077a into Mesh-LLM:main May 24, 2026
18 checks passed
michaelneale added a commit that referenced this pull request May 24, 2026
* origin/main:
  Honor direct chat no-think requests (#647)
  fix(swift-sdk): produce versioned bundle layout for macOS xcframework (#665)
  Update llama.cpp upstream pin
  Fix native SDK runtime crate packaging (#664)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants