[Mistral] Move MistralConverter into integrations/mistral/ package by juliendenize · Pull Request #46603 · huggingface/transformers

juliendenize · 2026-06-12T13:56:48Z

What does this PR do?

Move the tekken→HF tokenizer converter into a dedicated, lazily-loaded Mistral integration package.
Fix the vocab index offset so special tokens occupy the first ids.
Add from_tekken_file(), which pre-computes the vocab and merges in a single pass.
Assign special-token ids from their explicit rank, independent of list ordering.
Register special tokens as special so they are stripped when decoding with skip_special_tokens=True.
Preserve the original tekken metadata so the tokenizer can be re-saved losslessly.

Tested with RUN_SLOW=1 pytest tests/integrations/mistral/test_tokenizer.py → 22 passed (incl. slow tests downloading tekken.json from mistralai/Ministral-3-3B-Instruct-2512 and mistral-common cross-backend equivalence).

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

tokenizers: @ArthurZucker and @itazap

HuggingFaceDocBuilderDev · 2026-06-16T08:56:51Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ArthurZucker

Ty! Have 1 small nit

github-actions · 2026-06-22T22:33:26Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-06-22T22:35:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: afmoe, aimv2, albert, align, altclip, apertus, arcee, aria, audio_spectrogram_transformer, audioflamingo3, auto, autoformer, aya_vision, bamba, bark, bart

juliendenize · 2026-06-23T13:19:21Z

@ArthurZucker hopefully i answered to your comments 😄

itazap · 2026-07-08T10:58:04Z

+        return instance
+
+    @staticmethod
+    def _extract_merges(bpe_ranks: dict[bytes, int]) -> tuple[dict[str, int], list[tuple[str, str]]]:


copy of TikTokenConverter.extract_vocab_merges_from_model (https://github.com/juliendenize/transformers/blob/b4765e266167036650168e1baaadfd2ce1138a6c/src/transformers/convert_slow_tokenizer.py#L1921) so maybe we should actually have this as a top-level function in convert_slow_tokenizer.py that can be imported by both MistralConverter and TikTokenConverter.

I can do this in a follow up PR though, to keep this one mistral-only

makes sense, think it would be better in a follow up as well for the reason you invoked

…rd mistral-common import

github-actions · 2026-07-08T14:34:52Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28948624437:2
Result: success | Jobs: 15 | Tests: 172,394 | Failures: 0 | Duration: 7h 27m

itazap

Awesome, thanks!

…uggingface#46603) * [Mistral] Move MistralConverter into integrations/mistral/ package * refactor: make MistralConverter constructible from vocab_file and guard mistral-common import

juliendenize mentioned this pull request Jun 12, 2026

[Mistral] Add tekken tokenizer detection, conversion, and save utilities #46604

Closed

6 tasks

juliendenize force-pushed the mistral/01-package-converter branch 2 times, most recently from 000d773 to ca2d153 Compare June 12, 2026 14:10

juliendenize marked this pull request as ready for review June 12, 2026 14:24

github-actions Bot requested review from SunMarc and ydshieh June 12, 2026 14:24

SunMarc requested review from ArthurZucker and itazap June 15, 2026 15:36

ArthurZucker approved these changes Jun 18, 2026

View reviewed changes

Comment thread src/transformers/integrations/mistral/__init__.py

Comment thread src/transformers/integrations/mistral/__init__.py

Comment thread tests/integrations/mistral/test_tokenizer.py Outdated

Comment thread src/transformers/tokenization_utils_tokenizers.py

juliendenize force-pushed the mistral/01-package-converter branch 3 times, most recently from f23ce60 to 675280e Compare June 22, 2026 22:33

juliendenize force-pushed the mistral/01-package-converter branch 2 times, most recently from 3440908 to f3bd1d9 Compare June 22, 2026 22:59

[Mistral] Move MistralConverter into integrations/mistral/ package

87f2b6d

juliendenize force-pushed the mistral/01-package-converter branch from f3bd1d9 to 87f2b6d Compare June 25, 2026 14:31

juliendenize added 4 commits June 26, 2026 16:43

Merge branch 'main' into mistral/01-package-converter

2a610db

Merge branch 'main' into mistral/01-package-converter

6903cc0

Merge branch 'main' into mistral/01-package-converter

7e6a617

Merge branch 'main' into mistral/01-package-converter

b4765e2

itazap reviewed Jul 8, 2026

View reviewed changes

Comment thread src/transformers/integrations/mistral/tokenizer.py Outdated

itazap reviewed Jul 8, 2026

View reviewed changes

Comment thread src/transformers/integrations/mistral/tokenizer.py

itazap reviewed Jul 8, 2026

View reviewed changes

refactor: make MistralConverter constructible from vocab_file and gua…

fb89110

…rd mistral-common import

itazap approved these changes Jul 9, 2026

View reviewed changes

ArthurZucker added this pull request to the merge queue Jul 17, 2026

Merged via the queue into huggingface:main with commit ab78dc9 Jul 17, 2026
106 checks passed

juliendenize mentioned this pull request Jul 23, 2026

[Mistral] Add native tekken tokenizer support to AutoTokenizer #47507

Merged

6 tasks

juliendenize mentioned this pull request Aug 7, 2026

[Mistral] Add params.json and HF config conversion #47835

Open

6 tasks

Navigation Menu

Uh oh!

[Mistral] Move MistralConverter into integrations/mistral/ package - #46603

[Mistral] Move MistralConverter into integrations/mistral/ package#46603
ArthurZucker merged 6 commits into
huggingface:mainfrom
juliendenize:mistral/01-package-converter

juliendenize commented Jun 12, 2026 •

edited by github-actions Bot

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Jun 16, 2026

Uh oh!

ArthurZucker left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

juliendenize commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

itazap Jul 8, 2026

Uh oh!

juliendenize Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

itazap left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

juliendenize commented Jun 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jun 16, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

juliendenize commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

itazap Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

juliendenize Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 8, 2026

CI recap

Uh oh!

itazap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

juliendenize commented Jun 12, 2026 •

edited by github-actions Bot

Loading