fix: drop chat_template_kwargs for Mistral 3.2 compatibility

[EnjoyBacon7]

Summary

Mistral-Small-3.2-24B-Instruct-2506-FP8 ships native tokenizer files (tekken.json / params.json), causing vLLM to load mistral_common which rejects any chat_template field — including chat_template_kwargs in the request body (ValueError: chat_template is not supported for Mistral tokenizers).

Remove extra_body: {"chat_template_kwargs": {"enable_thinking": False}} from the pipeline, base loader, and benchmark scripts.

Test plan

• POST /v1/chat/completions no longer raises the ValueError
• Tool calling and image captioning still work

[coderabbitai]

📝 Walkthrough

Walkthrough

Across four files in benchmarks and component modules, conditional extra_body parameters that disabled extended thinking for non-OpenAI endpoints have been commented out. Model initialization now omits explicit thinking disable configuration while retaining other parameters like max_completion_tokens.

Changes

Cohort / File(s)	Summary
Benchmark Model Configuration benchmarks/prompt_eval/eval_query_decomposition.py, benchmarks/prompt_eval/eval_temporal_filter_generation.py	Removed conditional extra_body parameter injection for vLLM-specific thinking disable; _model_kwargs helper now returns only max_completion_tokens.
Component Model Initialization openrag/components/indexer/loaders/base.py, openrag/components/pipeline.py	Commented out extra_body.chat_template_kwargs.enable_thinking=False in ChatOpenAI and query generator binding; models now initialized without explicit thinking disable configuration.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 No more thinking to suppress,
The extra_body's been laid to rest,
With max_tokens light and lean,
The cleanest config I have seen! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: removing chat_template_kwargs configuration across multiple files for Mistral 3.2 compatibility.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/drop-chat-template-kwargs-mistral-3.2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (3)

benchmarks/prompt_eval/eval_temporal_filter_generation.py (1)

232-234: Duplicate of the change in eval_query_decomposition.py.

Same note: prefer deletion or an explicit config flag over a long-lived commented block. Since the helper is now trivial, the base_url parameter is also unused — consider inlining {"max_completion_tokens": 512} at call sites or dropping the parameter.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@benchmarks/prompt_eval/eval_temporal_filter_generation.py` around lines 232 -
234, Remove the long-lived commented block and the now-unused base_url parameter
from the helper in benchmarks/prompt_eval/eval_temporal_filter_generation.py
(the same change applied in eval_query_decomposition.py): delete the commented
`if "openai.com" ...` lines, remove `base_url` from the helper signature, and
either inline the existing `{"max_completion_tokens": 512}` at its call sites or
add an explicit config flag if you need toggling; update all callers to stop
passing base_url or to use the inlined default.

openrag/components/indexer/loaders/base.py (1)

41-47: LGTM — minor: drop the commented line.

The VLM captioning path doesn't benefit from thinking mode anyway, so removing enable_thinking=False is safe for the Mistral case. Suggest deleting the commented entry (Line 45) rather than keeping dead code in model_settings.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@openrag/components/indexer/loaders/base.py` around lines 41 - 47, Remove the
dead commented entry from the model_settings dict: delete the commented line "#
"extra_body": {"chat_template_kwargs": {"enable_thinking": False}}," so
model_settings only contains active keys ("temperature", "max_retries",
"timeout") and then continue to update settings as before (refer to the
model_settings variable and the subsequent settings.update(model_settings)
call).

benchmarks/prompt_eval/eval_query_decomposition.py (1)

203-205: Consider removing the dead commented block rather than leaving it in the source.

Commented-out code tends to rot. If the intent is to re-enable enable_thinking=False for some future endpoint, gate it behind an env var (e.g., DISABLE_THINKING=1) or model-config flag instead of preserving commented lines. Otherwise, delete them — git history is authoritative.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@benchmarks/prompt_eval/eval_query_decomposition.py` around lines 203 - 205,
Remove the dead commented-out conditional that checks "openai.com" and sets
kwargs["extra_body"] = {"chat_template_kwargs": {"enable_thinking": False}};
either delete those commented lines entirely from eval_query_decomposition.py,
or if you actually need the behavior, implement it properly by reading an
environment flag (e.g., DISABLE_THINKING) or a model-config flag and, before the
existing return kwargs, set kwargs["extra_body"] = {"chat_template_kwargs":
{"enable_thinking": False}} when the flag is present, using the existing
base_url/kwargs variables (do not leave commented code).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@benchmarks/prompt_eval/eval_query_decomposition.py`:
- Around line 203-205: Remove the dead commented-out conditional that checks
"openai.com" and sets kwargs["extra_body"] = {"chat_template_kwargs":
{"enable_thinking": False}}; either delete those commented lines entirely from
eval_query_decomposition.py, or if you actually need the behavior, implement it
properly by reading an environment flag (e.g., DISABLE_THINKING) or a
model-config flag and, before the existing return kwargs, set
kwargs["extra_body"] = {"chat_template_kwargs": {"enable_thinking": False}} when
the flag is present, using the existing base_url/kwargs variables (do not leave
commented code).

In `@benchmarks/prompt_eval/eval_temporal_filter_generation.py`:
- Around line 232-234: Remove the long-lived commented block and the now-unused
base_url parameter from the helper in
benchmarks/prompt_eval/eval_temporal_filter_generation.py (the same change
applied in eval_query_decomposition.py): delete the commented `if "openai.com"
...` lines, remove `base_url` from the helper signature, and either inline the
existing `{"max_completion_tokens": 512}` at its call sites or add an explicit
config flag if you need toggling; update all callers to stop passing base_url or
to use the inlined default.

In `@openrag/components/indexer/loaders/base.py`:
- Around line 41-47: Remove the dead commented entry from the model_settings
dict: delete the commented line "# "extra_body": {"chat_template_kwargs":
{"enable_thinking": False}}," so model_settings only contains active keys
("temperature", "max_retries", "timeout") and then continue to update settings
as before (refer to the model_settings variable and the subsequent
settings.update(model_settings) call).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6d8f0686-c76f-4755-9c4e-ded1df521aea

📥 Commits

Reviewing files that changed from the base of the PR and between d27c27b and 2f2bf2e.

📒 Files selected for processing (4)

• benchmarks/prompt_eval/eval_query_decomposition.py
• benchmarks/prompt_eval/eval_temporal_filter_generation.py
• openrag/components/indexer/loaders/base.py
• openrag/components/pipeline.py