fix(generation): Enforce the auto-compile cache check for encoder-decoder models by harshaljanjani · Pull Request #48364 · huggingface/transformers

harshaljanjani · 2026-08-27T07:55:57Z

What does this PR do?

→ Follow-up to this comment on the Canary PR; the encoder-decoder cache holds two subcaches and the auto-compile check isn't propagated through them.
→ No behaviour change on main

cc: @ebezzam @vasqu

Code Agent Policy

I confirm that this is not a pure code agent PR.

Before submitting

Did you read the contributor guideline, Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you fix any necessary existing tests?

vasqu

Also cc @Cyrilvallez and @zucchini-nlp for generation in general

vasqu · 2026-08-27T08:27:18Z

+        # layers are, but we don't want to ALWAYS compile when calling `generate`, so we check the type. Encoder-decoder
+        # models hold that cache in a subcache, so we unwrap it to check the cache the decoder actually generates with
+        decoder_cache = cache.self_attention_cache if isinstance(cache, EncoderDecoderCache) else cache
+        using_compilable_cache = cache is not None and cache.is_compileable and type(decoder_cache) is not DynamicCache


Suggested change

# layers are, but we don't want to ALWAYS compile when calling `generate`, so we check the type. Encoder-decoder

# models hold that cache in a subcache, so we unwrap it to check the cache the decoder actually generates with

decoder_cache = cache.self_attention_cache if isinstance(cache, EncoderDecoderCache) else cache

using_compilable_cache = cache is not None and cache.is_compileable and type(decoder_cache) is not DynamicCache

# Encoder-decoder models hold that cache in a subcache, so we unwrap it to check the cache the decoder actually generates with

decoder_cache = cache.self_attention_cache if isinstance(cache, EncoderDecoderCache) else cache

# Note: for some models that only use linear attention (e.g. Mamba), even a DynamicCache is compilable since all

# layers are, but we don't want to ALWAYS compile when calling `generate`, so we check the type

using_compilable_cache = cache is not None and cache.is_compileable and type(decoder_cache) is not DynamicCache

let's split the comments per line more readable imo

isn't checking cache.is_compileable enough in general, i don't think we have compileable Dynamic Cache 🤔

Check the comment above (lin attn caches)

Resolved, thank you both :)

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

vasqu

Small last comments from my side

vasqu · 2026-08-31T12:19:20Z

+    def test_compileable_default_cache_doesnt_compile_encoder_decoder(self):
+        """Test that a compileable default cache doesn't trigger compilation on encoder-decoder models either"""
+        model = AutoModelForSeq2SeqLM.from_pretrained("hf-internal-testing/tiny-random-bart")
+        decoder_config = copy.deepcopy(model.config.get_text_config(decoder=True))


Might be overkill since it only lives for this test

You're right, dropped it :)

vasqu · 2026-08-31T12:19:52Z

+        )
+        generation_config = GenerationConfig()
+        generation_config.compile_config = CompileConfig()
+        generation_config.compile_config._compile_all_devices = True


Just wondering, could we set it at construction time?

Can't unfortunately, it's a normal attr not a dataclass field in CompileConfig. Set it like the other usages.

vasqu · 2026-08-31T12:23:21Z

+        generation_config = GenerationConfig()
+        generation_config.compile_config = CompileConfig()
+        generation_config.compile_config._compile_all_devices = True
+        self.assertFalse(model._valid_auto_compile_criteria({"past_key_values": cache}, generation_config))


I guess it needs the static cache to be true?

Yup because otherwise the test passes on CPU regardless. Static cache alone doesn't repro the issue either (both main and the PR return True). The actual diff in behavior is with a compileable DynamicCache (main returns True but the fix returns False) so that's the behavior we're trying to catch.

vasqu

Nice thanks for the explanations, let's merge

HuggingFaceDocBuilderDev · 2026-08-31T14:14:05Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

vasqu · 2026-08-31T14:23:23Z

Hmm the failing test seems unrelated to me lemme try to rerun CI later today again

github-actions · 2026-08-31T14:45:11Z

CI recap

Dashboard: View test results in Grafana
Latest run: 33395774991:4
Result: success | Jobs: 1 | Tests: 29 | Failures: 0 | Duration: 6m 20s

vasqu · 2026-08-31T14:53:19Z

Syncing with main worked 🤷 should merge soon

…oder models (huggingface#48364) * fix: Enforce the auto-compile cache check for encoder-decoder models * change: Update generation/utils.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * fix: Changes --------- Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

fix: Enforce the auto-compile cache check for encoder-decoder models

274afcf

harshaljanjani mentioned this pull request Aug 27, 2026

model: Add NVIDIA Canary-1B-v2 to Transformers #46825

Merged

4 tasks done

vasqu reviewed Aug 27, 2026

View reviewed changes

change: Update generation/utils.py

0cff6c1

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

harshaljanjani requested a review from vasqu August 27, 2026 10:50

vasqu approved these changes Aug 31, 2026

View reviewed changes

fix: Changes

5a3c5ba

harshaljanjani requested a review from vasqu August 31, 2026 13:25

vasqu approved these changes Aug 31, 2026

View reviewed changes

vasqu enabled auto-merge August 31, 2026 14:04

Merge branch 'main' into fix/enc-dec-cache-compile-criteria

6628339

vasqu added this pull request to the merge queue Aug 31, 2026

Merged via the queue into huggingface:main with commit e37b269 Aug 31, 2026
115 checks passed

Conversation

harshaljanjani commented Aug 27, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Aug 31, 2026

Uh oh!

vasqu commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

CI recap

Uh oh!

vasqu commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

harshaljanjani commented Aug 27, 2026 •

edited by github-actions Bot

Loading