Add dots3-note Preview model support by miraclezqc · Pull Request #47844 · huggingface/transformers

miraclezqc · 2026-08-08T13:10:17Z

What does this PR do?

This PR is submitted by the official Dots-Studio team.

This PR adds native Transformers support for Dots 3 Note Preview, an inference-only mixture-of-experts multimodal causal language model supporting text, image, video, and audio inputs.

The implementation follows the modular Transformers workflow and includes the model configuration, processor, image processor, audio feature extractor, modeling implementation, auto-class registrations, documentation, conversion utilities, and model tests required for native integration.

The implementation includes:

text, image, video, and audio input processing;
greedy multimodal generation;
sliding-window attention and DSA indexer inference;
multimodal cache handling during generation;
variable-length left-padded batching;
BF16 checkpoint support;
fine-grained FP8 checkpoint support using dynamic activations, E4M3 weights, and a 128 x 128 weight block size;
FP32 routing for the vision MoE router.

This integration currently targets inference. Training-specific features such as gradient checkpointing are outside the scope of this PR.

No additional runtime dependency is introduced by this integration.

Validation

The following checks were run against the implementation in this PR:

the complete tests/models/dots3_note test directory;
modular-to-generated modeling conversion and synchronization checks;
Ruff formatting and lint checks;
repository diff checks;
clean loading of the real BF16 and FP8 checkpoints;
greedy end-to-end generation through AutoModelForMultimodalLM;
text reasoning tests, including GSM8K-style problems;
speech transcription;
image question answering;
video understanding;
multimodal cache and masking regression tests;
FP8 dequantization with non-divisible weight dimensions;
DSA projection precision and derived-weight cache invalidation;
vision MoE near-tie routing precision.

Both the BF16 and FP8 checkpoints passed the end-to-end text, image, video, and audio validation cases with finite outputs and successful greedy generation.

The corresponding public checkpoint links and finalized model organization metadata will be added after this PR is merged.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. These often are low-quality, or fix extremely minor issues that occur rarely or never in practice.
As a result, we're instituting a rule that first-time contributors should not use code agents to submit PRs or issues.
We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues.

Issues/PRs from first-time contributors that violate this rule will probably just be closed without review, and we
might block you, especially if you open more than one or appear to be deliberately ignoring this. We especially do not
want new contributors to jump in on random issues to contribute an agent-written fix. This creates lots of noise
for reviewers and other users and will almost certainly get you blocked.

For more information, please read CONTRIBUTING.md.

(First-time contributors only): I confirm that this PR description and code is not written by an LLM or code agent

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline and the Pull Request checks?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes according to the guidelines?
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed.

cc @zucchini-nlp @vasqu

zucchini-nlp · 2026-08-13T09:54:53Z

hey @miraclezqc , thanks for the PR! Most of the team currently is off on summer vacations so the reviews are taking longer. We'll review it soon :)

miraclezqc · 2026-08-13T10:46:35Z

hey @miraclezqc , thanks for the PR! Most of the team currently is off on summer vacations so the reviews are taking longer. We'll review it soon :)

@zucchini-nlp
Thanks for the update—we completely understand and look forward to your comments when the team has time to review!

ArthurZucker · 2026-08-18T09:49:13Z

Hey! Happy to review, can you update the branch to fix the latest mlinter updates!

miraclezqc · 2026-08-18T12:49:24Z

Hey! Happy to review, can you update the branch to fix the latest mlinter updates!

@ArthurZucker

Thanks! The branch has been updated for the latest mlinter rules.

It seems that the new CI run is blocked by the security gate. Could you please approve it?

HuggingFaceDocBuilderDev · 2026-08-20T08:03:30Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

miraclezqc · 2026-08-24T05:35:42Z

@ArthurZucker Hi. The branch is now up to date with main and all local checks pass.
But the security gate is blocked again. Could you please approve the workflow and trigger the suggested slow jobs?

github-actions · 2026-08-24T08:26:08Z

CI recap

Dashboard: View test results in Grafana
Latest run: 32138376283:2
Result: success | Jobs: 16 | Tests: 182,471 | Failures: 1 | Duration: 12h 28m

miraclezqc · 2026-08-25T02:54:03Z

Thanks for approving the workflow! All regular CI checks are now green, and the branch is ready for review.
We’d appreciate your review when convenient and will address any feedback promptly.

miraclezqc · 2026-08-27T04:12:36Z

Hi @ArthurZucker @Rocketknight1 , the branch is now up to date with main, and all relevant local tests pass. We’d appreciate your review and an early merge if everything looks good. Thank you!

vasqu · 2026-09-04T14:14:12Z

Sorry for the delays @miraclezqc but I noticed that we are not using modular as intended at all, please check out https://huggingface.co/docs/transformers/modular_transformers

The goal is to reuse existing modules as much as we can, this includes the rms norm, indexer, etc

miraclezqc · 2026-09-05T06:40:39Z

@vasqu Thanks! We’ve refactored the model to reuse existing DeepSeek, GLM DSA, and Qwen2-VL components, including RMSNorm and the indexer. Only Dots-specific logic remains custom, and all relevant tests pass.

Could you please take another look and let us know if it is ready to merge?

vasqu

I tried to go a bit more into details but this is still not super modular friendly: There are a lot of custom implementations where we could just reuse ours - e.g. the dsa is handled completely custom, the attn backend is also completely custom which would disallow any usage via vllm, etc.

What I want to tell with this: Custom implementations are very much still implemented in transformers oftentimes but they need to be aligned - imo the biggest example is RoPE in this case

vasqu · 2026-09-07T18:15:12Z

+> [!NOTE]
+> Loading encoded audio or video sources requires the optional `torchcodec` dependency. Native video preprocessing
+> follows the training-time sampling pipeline and expands each video into timestamped image blocks with interleaved
+> audio blocks. The `<|video_pad|>` marker is only an external prompt placeholder and is removed before tokenization;
+> it is never passed to the model. Decoded frame arrays are supported as visual-only inputs because their original
+> audio track is absent.


I think this is fairly common across all of our models so I think we can delete it but no strong opinion

vasqu · 2026-09-07T18:15:37Z

+> it is never passed to the model. Decoded frame arrays are supported as visual-only inputs because their original
+> audio track is absent.
+
+```python


missing usage header?

vasqu · 2026-09-07T18:15:51Z

+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"},


Also might be nice to show each modality wdyt?

vasqu · 2026-09-07T18:16:42Z

    return model


+def _resolve_weight_block_size(hf_quantizer, value: torch.Tensor) -> tuple[int, int]:


this looks unnecessary no? Can we revert to keep it inline and not a separate fn?

vasqu · 2026-09-07T18:18:16Z

cc @IlyasMoutawwakil when you have time to check this over

can you share the motivation here? Ig there is some (new) fp4 handling we need?

vasqu · 2026-09-07T18:52:46Z

+    pass
+
+
+class Dots3NoteVisionRotaryEmbedding(nn.Module):


check qwen2 vl rope

vasqu · 2026-09-07T18:53:40Z

+    return attention_output, attention_weights
+
+
+class Dots3NoteVisionAttention(nn.Module):


same here a lot of custom stuff we should not use and try to align with existing models

vasqu · 2026-09-07T18:54:12Z

+# Unified multimodal model
+# -----------------------------------------------------------------------------
+@auto_docstring
+class Dots3NoteForCausalLM(Dots3NoteTextForCausalLM):


ForConditional - pls check the pattern in qwen vl

or gemma4 and the like (omni)

vasqu · 2026-09-07T18:55:22Z

+
+
+@auto_docstring
+class Dots3NoteForConditionalGeneration(Dots3NoteForCausalLM):


Doesnt make much sense to make for causal lm and for conditional

Is the goal to have a text only variation, then pls check out qwen3_5 which does that

vasqu · 2026-09-07T18:56:00Z

Modular can also be applied for all processor related things

Didnt go through these but we have a lot of custom things we shouldnt need

github-actions · 2026-09-09T06:20:27Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/ — this covers files the PR deletes or renames, not only the ones it edits
A new high-severity security issue was detected in the changed Python files (Bandit check)
The PR touches a path this repository protects from untrusted PRs, such as the file that decides who reviews it — a maintainer must make that change in a separate PR

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-09-09T06:21:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, dots3_note, finegrained_fp8

miraclezqc · 2026-09-09T06:28:08Z

I tried to go a bit more into details but this is still not super modular friendly: There are a lot of custom implementations where we could just reuse ours - e.g. the dsa is handled completely custom, the attn backend is also completely custom which would disallow any usage via vllm, etc.

What I want to tell with this: Custom implementations are very much still implemented in transformers oftentimes but they need to be aligned - imo the biggest example is RoPE in this case

Thanks @vasqu for the detailed review and all the helpful suggestions. We’ve gone through the comments and refactored the text, audio, vision, and preprocessing code to reuse existing Transformers components, removing redundant implementations along the way.

We believe most suggestions are addressed. The remaining differences and follow-ups are:

Shared MoE change: removed from this PR as requested. We will make a separate PR for this.
Shared FP8 changes: the remaining changes handle partial weight blocks, rather than adding FP4 support, and preserve converter scope/prefix information while handling already-anchored weight patterns. These are separate from the model-specific refactor.
Indexer: the custom projections and scoring implementation are removed in favor of GLM’s implementation. Loading-time dequantization of the original FP8 indexer weights/scales remains TODO.
Text attention: the custom forward retains Dots’ LoRA scaling, separate K-RoPE normalization, and output sigmoid gate, which are absent from the DeepSeek-V3.2 forward. The shared projections, KV expansion, and attention interface are reused.
Initialization and output handling: a small initializer remains for the per-layer-type RoPE buffers not restored by the DeepSeek implementation. Some model declarations and explicit attention/vision output handling also remain to preserve supported configurations and the existing output structure.
Model naming: text-only and multimodal classes are separated. The old Dots3NoteForCausalLM name remains only for compatibility with the original checkpoints, without a duplicate forward.
Preprocessing: image/video processing reuses Qwen components. Native-video audio/frame interleaving and joint token budgeting remain model-specific.

We may still have missed a comment or a simpler reuse opportunity. Could you take another look and let us know what else needs to change before merging? Thanks again for your time and guidance.

vasqu · 2026-09-10T18:23:32Z

Checking tomorrow! 🤗

miraclezqc added 4 commits August 7, 2026 20:29

Add Dots3-Note Omni model support

4271569

Fix Dots3-Note Omni DSA and FP8 loading

ee04344

Fix Dots3-Note Omni masking and multimodal caching

e4b8919

Fix Dots3-Note Omni inference precision and FP8 config

b5cbe05

miraclezqc marked this pull request as ready for review August 8, 2026 13:21

github-actions Bot requested review from ArthurZucker and Rocketknight1 August 8, 2026 13:22

miraclezqc changed the title ~~Add dots3 note omni~~ Add Dots3-Note Omni model support Aug 8, 2026

miraclezqc added 3 commits August 8, 2026 14:24

Fix Dots3-Note Omni import structure

bdefaaa

Fix Dots3-Note Omni multimodal preprocessing

50d7cb0

Fix Dots3-Note Omni configuration and CI checks

fe87862

miraclezqc force-pushed the add-dots3-note-omni branch from 7af5a25 to fe87862 Compare August 9, 2026 05:45

Rename Dots3-Note Omni to Dots 3 Note Preview

a7eafce

miraclezqc changed the title ~~Add Dots3-Note Omni model support~~ Add Dots 3 Note Preview model support Aug 10, 2026

Fix Dots 3 Note Preview processor defaults and video budgets

d8e147f

miraclezqc force-pushed the add-dots3-note-omni branch from b70c2d9 to d8e147f Compare August 13, 2026 10:27

Fix Dots 3 Note Preview checkpoint references and tests

9daa866

miraclezqc force-pushed the add-dots3-note-omni branch from 84acd71 to 9daa866 Compare August 13, 2026 20:06

miraclezqc changed the title ~~Add Dots 3 Note Preview model support~~ Add dots3-note Preview model support Aug 14, 2026

ngxson mentioned this pull request Aug 14, 2026

model: add dots3-note ggml-org/llama.cpp#27060

Merged

4 tasks

zucchini-nlp and others added 2 commits August 18, 2026 12:31

Merge branch 'main' into add-dots3-note-omni

e76affc

Fix Dots 3 Note Preview modeling checks

1f884b3

miraclezqc added 3 commits August 20, 2026 10:01

Fix Dots 3 Note Preview model date

59e063f

Merge branch 'main' into add-dots3-note-omni

45c24f2

Fix Dots 3 Note Preview compatibility with main

f8a1f00

Merge branch 'main' into add-dots3-note-omni

3732a1d

Merge branch 'main' into add-dots3-note-omni

d8acaab

miraclezqc added 2 commits September 5, 2026 06:19

Merge branch 'main' into add-dots3-note-omni

db923d7

Refactor Dots 3 Note Preview with modular components

42974dc

vasqu reviewed Sep 7, 2026

View reviewed changes

miraclezqc added 2 commits September 9, 2026 06:01

Refactor Dots 3 Note Preview to reuse multimodal components

c6bdaed

Merge branch 'main' into add-dots3-note-omni

cc8c2f3

		return model


		def _resolve_weight_block_size(hf_quantizer, value: torch.Tensor) -> tuple[int, int]:

		return attention_output, attention_weights


		class Dots3NoteVisionAttention(nn.Module):



		@auto_docstring
		class Dots3NoteForConditionalGeneration(Dots3NoteForCausalLM):

Uh oh!

Conversation

miraclezqc commented Aug 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Validation

Code Agent Policy

Before submitting

Who can review?

Uh oh!

zucchini-nlp commented Aug 13, 2026

Uh oh!

miraclezqc commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArthurZucker commented Aug 18, 2026

Uh oh!

miraclezqc commented Aug 18, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Aug 20, 2026

Uh oh!

miraclezqc commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026

CI recap

Uh oh!

miraclezqc commented Aug 25, 2026

Uh oh!

miraclezqc commented Aug 27, 2026

Uh oh!

vasqu commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miraclezqc commented Sep 5, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 9, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

miraclezqc commented Sep 9, 2026

Uh oh!

vasqu commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

miraclezqc commented Aug 8, 2026 •

edited by github-actions Bot

Loading

miraclezqc commented Aug 13, 2026 •

edited

Loading

vasqu commented Sep 4, 2026 •

edited

Loading