Skip to content

[Misc] Add a Jinja template to support Mistral3 function calling#17195

Merged
vllm-bot merged 1 commit intovllm-project:mainfrom
chaunceyjiang:mistral_init_kwargs
Apr 29, 2025
Merged

[Misc] Add a Jinja template to support Mistral3 function calling#17195
vllm-bot merged 1 commit intovllm-project:mainfrom
chaunceyjiang:mistral_init_kwargs

Conversation

@chaunceyjiang
Copy link
Collaborator

@chaunceyjiang chaunceyjiang commented Apr 25, 2025 •
edited by github-actions bot
Loading

Fix #16292

Usage:

vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 --chat-template examples/tool_chat_template_mistral3.jinja --tool-call-parser mistral --enable-auto-tool-choice

@github-actions
Copy link

github-actions bot commented Apr 25, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@chaunceyjiang chaunceyjiang marked this pull request as ready for review April 25, 2025 15:15
@chaunceyjiang chaunceyjiang changed the title [Bugfix] Fix 'MistralTokenizer' object has no attribute 'init_kwargs' [WIP][Bugfix] Fix 'MistralTokenizer' object has no attribute 'init_kwargs' Apr 25, 2025
DarkLight1337
DarkLight1337 reviewed Apr 25, 2025
View reviewed changes
@njhill njhill added the bug Something isn't working label Apr 25, 2025
@njhill njhill added this to the v0.8.5 milestone Apr 25, 2025
@chaunceyjiang chaunceyjiang force-pushed the mistral_init_kwargs branch 2 times, most recently from e9aec62 to 01ea1d2 Compare April 26, 2025 02:32
DarkLight1337
DarkLight1337 approved these changes Apr 26, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing!

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) April 26, 2025 03:25
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 26, 2025
@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 26, 2025

@chaunceyjiang can you confirm whether I can merge this PR? Since you added WIP to the title

@chaunceyjiang
Copy link
Collaborator Author

chaunceyjiang commented Apr 26, 2025

@DarkLight1337 I currently have no way to run tests because the GPU server I rented has expired.

Could you help me with the testing?

Also, I previously saw on Zhihu that joining the vLLM team might allow access to some GPU resources.
I'm wondering if I could join the vLLM team.

@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 26, 2025 •
edited
Loading

The test fails because that model repo (https://huggingface.co/RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16/tree/main) isn't in Mistral format.

@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 26, 2025

Also, I previously saw on Zhihu that joining the vLLM team might allow access to some GPU resources.
I'm wondering if I could join the vLLM team.

I don't have access to such resources. Probably, only the core members can use those machines.

auto-merge was automatically disabled April 26, 2025 13:40

Head branch was pushed to by a user without write access

@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 27, 2025

Given your comment in the original issue, I guess we can close this PR then?

@DarkLight1337 DarkLight1337 removed the ready ONLY add when PR is ready to merge/full CI is needed label Apr 27, 2025
@DarkLight1337 DarkLight1337 removed this from the v0.8.5 milestone Apr 28, 2025
@chaunceyjiang
Copy link
Collaborator Author

chaunceyjiang commented Apr 28, 2025

Given your comment in the original issue, I guess we can close this PR then?

Yes. I suggest submitting the Jinja templates tto the examples directory. What do you think?
@DarkLight1337

@DarkLight1337
Copy link
Member

DarkLight1337 commented Apr 28, 2025

That sounds good

@mergify mergify bot added documentation Improvements or additions to documentation tool-calling labels Apr 28, 2025
@chaunceyjiang
[Misc] Add a Jinja template to support Mistral3
06d480d 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
@chaunceyjiang chaunceyjiang changed the title [WIP][Bugfix] Fix 'MistralTokenizer' object has no attribute 'init_kwargs' [Misc] Add a Jinja template to support Mistral3 Apr 28, 2025
@chaunceyjiang chaunceyjiang changed the title [Misc] Add a Jinja template to support Mistral3 [Misc] Add a Jinja template to support Mistral3 function calling Apr 29, 2025
@vllm-bot vllm-bot merged commit 96e06e3 into vllm-project:main Apr 29, 2025
24 checks passed
@maxdebayser
Copy link
Contributor

maxdebayser commented Apr 29, 2025

Thanks for contributing! There seem to be just a few non-backward compatible changes compared to tool_chat_template_mistral_parallel.jinja. I couple of month ago I opened an issue to propose adding contextual information to the jinja global variables so that the chat template logic can adapt itself to the current model: #8869 . With this we would able to mitigate the proliferation of chat templates.

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@chaunceyjiang @lk-chen
[Misc] Add a Jinja template to support Mistral3 function calling (vll…
7df8cc1 
…m-project#17195)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@chaunceyjiang
[Misc] Add a Jinja template to support Mistral3 function calling (vll…
f9c79a4 
…m-project#17195)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025
@chaunceyjiang
[Misc] Add a Jinja template to support Mistral3 function calling (vll…
ffe926d 
…m-project#17195)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96

Assignees

No one assigned

Labels

bug Something isn't working documentation Improvements or additions to documentation tool-calling

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: MistralTokenizer not working when using Mistral Small 3.1 in HF format

5 participants

@chaunceyjiang @DarkLight1337 @maxdebayser @njhill @vllm-bot