[Whisper] handle deprecation of `forced_decoder_ids` #38232

gante · 2025-05-20T14:20:54Z

What does this PR do?

forced_decoder_ids, an old whisper flag, was deprecated in favor of the more explicit task and language flags (#28687). However, we need to handle it for BC: the original checkpoints have them.

This PR does a few deprecation-related changes, in an attempt to move us further away from using this flag:

Removes as many uses of generation_config.forced_decoder_ids as possible, except to handle BC in whisper's generate.
If the newer flags (task/language) are set, don't throw any warning, i.e. the deprecated flag is ignored if the newer flags are set. Users who loaded base whisper and applied these new flags were seeing warnings, and they shouldn't.
Don't mutate generation_config.forced_decoder_ids, and ignore it in GenerationMixin.generate

✅ no regressions in SLOW whisper tests

gante · 2025-05-20T14:22:08Z

src/transformers/generation/configuration_utils.py

-
-        > Wild card
-
-        generation_kwargs:


note: this flag didn't have any uses in our codebase. I've added it in one of the first commits, but we added support to arbitrary kwargs a few weeks after that (for whisper)

Hmm, I see. Is it still needed in the future or just kept for BC; not 100% familiar yet.

In an ideal world, we would deprecate this sometime in the future imo. It makes it hard to know what is supported and what not.

not used at all in our library!

BC is not broken when this attribute exists in a file: we load arbitrary kwargs at init time :P The difference is that a) it is not documented b) self.generation_kwargs won't exist by default on instances (IMO not worth the extra work of a deprecation cycle for this detail)

Yea, not worth the effort (for now). Good to know tho!

src/transformers/generation/configuration_utils.py

src/transformers/models/whisper/generation_whisper.py

HuggingFaceDocBuilderDev · 2025-05-20T14:33:54Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

vasqu

LGTM! Mostly nits, functionality wise looks good :)

vasqu · 2025-05-20T15:35:50Z

src/transformers/generation/configuration_utils.py

-
-        > Wild card
-
-        generation_kwargs:


Hmm, I see. Is it still needed in the future or just kept for BC; not 100% familiar yet.

In an ideal world, we would deprecate this sometime in the future imo. It makes it hard to know what is supported and what not.

vasqu · 2025-05-20T15:37:19Z

src/transformers/generation/logits_process.py

        self,
-        generate_config,
-        begin_index: Optional[int] = None,
+        generate_config: "GenerationConfig",


Any reason we just don't typecheck by default? I.e. meaning

#if TYPE_CHECKING: from ..generation.configuration_utils import GenerationConfig

yeah -- it would result in a circular import

(technically there shouldn't be one, so at some point we need to clear things)

Oof, yea that's rough. Can you add a comment on the import? Otherwise, others (me) might forget again :D

src/transformers/generation/logits_process.py

vasqu · 2025-05-20T15:42:39Z

src/transformers/models/whisper/generation_whisper.py

+            forced_decoder_ids = getattr(generation_config, "forced_decoder_ids", None)
+            # fallback: check the model config for forced_decoder_ids
+            if forced_decoder_ids is None and getattr(config, "forced_decoder_ids", None) is not None:
+                forced_decoder_ids = config.forced_decoder_ids


Looks doubled, overwriting forced_decoder_ids. I doubt it's intended?

It's intentional: for BC reasons, we fallback generation_config attributes to the ones set in config.

The actual logic generate uses is more complex, this is a simplified version. Note that I didn't write these lines, I simply moved them around :)

Omg, i mistook config with generation_config - now it makes more sense. Yea, no worries - looks good (even if it's ugly through BC)

src/transformers/models/whisper/generation_whisper.py

vasqu

Thanks and good work!

vasqu · 2025-05-20T16:24:04Z

src/transformers/generation/configuration_utils.py

-
-        > Wild card
-
-        generation_kwargs:


Yea, not worth the effort (for now). Good to know tho!

vasqu · 2025-05-20T16:24:50Z

src/transformers/generation/logits_process.py

        self,
-        generate_config,
-        begin_index: Optional[int] = None,
+        generate_config: "GenerationConfig",


Oof, yea that's rough. Can you add a comment on the import? Otherwise, others (me) might forget again :D

vasqu · 2025-05-20T16:26:15Z

src/transformers/models/whisper/generation_whisper.py

+            forced_decoder_ids = getattr(generation_config, "forced_decoder_ids", None)
+            # fallback: check the model config for forced_decoder_ids
+            if forced_decoder_ids is None and getattr(config, "forced_decoder_ids", None) is not None:
+                forced_decoder_ids = config.forced_decoder_ids


Omg, i mistook config with generation_config - now it makes more sense. Yea, no worries - looks good (even if it's ugly through BC)

gante added 4 commits May 20, 2025 11:23

fix

8d98097

working saved forced_decoder_ids

dede7c0

docstring

8c874d7

add deprecation message

b920f50

gante requested a review from vasqu May 20, 2025 14:21

gante commented May 20, 2025

View reviewed changes

src/transformers/generation/configuration_utils.py Show resolved Hide resolved

gante commented May 20, 2025

View reviewed changes

src/transformers/models/whisper/generation_whisper.py Show resolved Hide resolved

vasqu reviewed May 20, 2025

View reviewed changes

exception message ordering

84d3d8d

vasqu approved these changes May 20, 2025

View reviewed changes

circular import comment

7fd9a2f

gante enabled auto-merge (squash) May 22, 2025 09:05

gante merged commit f8630c7 into huggingface:main May 22, 2025
20 checks passed

gante deleted the forced_decoder_ids branch May 22, 2025 09:16

gante mentioned this pull request May 23, 2025

[tf/flax] handle forced_decoder_ids deletion #38316

Merged

echarlaix mentioned this pull request Jun 11, 2025

Example from HuggingFace docs does not work with optimum 1.25.3 huggingface/optimum#2280

Closed

4 tasks

aleksandr-mokrov mentioned this pull request Aug 4, 2025

[OpenVINO] Handle deprecation of forced_decoder_ids in transformers generation_config huggingface/optimum-intel#1402

Merged

[Whisper] handle deprecation of forced_decoder_ids #38232

[Whisper] handle deprecation of forced_decoder_ids #38232

Conversation

gante commented May 20, 2025

What does this PR do?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gante May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented May 20, 2025

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Whisper] handle deprecation of `forced_decoder_ids` #38232

[Whisper] handle deprecation of `forced_decoder_ids` #38232

gante May 20, 2025 •

edited

Loading