Restrict VLM padding workaround to transformers 5.3.0

[albertvillanova]

Restrict VLM padding workaround to transformers 5.3.0.

This PR updates the prompt tokenization logic in several trainer classes to handle a specific bug in the transformers library more robustly. The main change is to apply a padding workaround only for affected transformers versions (5.3.x), and to simplify the logic for handling padded and unpadded input IDs. This avoids unnecessary padding for unaffected versions and ensures compatibility with future releases.

Additionally, this PR fixes:

• Warning log: Kwargs passed to processor.__call__ have to be in processor_kwargs dict, not in **kwargs #5502
• this issue arises when passing padding=True, but only with transformers >= 5.4.0

Fix #5502.

Motivation

• The bug (Qwen2_5_VLProcessor.apply_chat_template crashing on batched unpadded input) was introduced in transformers 5.3.0:

  • The original comment also incorrectly attributed the bug to transformers 5.2.0.
  • Qwen2_5_VLProcessor.apply_chat_template crashes on batched input when padding=False transformers#44514

• The bug was fixed in 5.4.0

  • Allow mm_token_type be non-padded lists  transformers#44563

• In trl, it was fixed with padding=True workaround in _tokenize_prompts (GRPO, RLOO, DPPO trainers) that was applied unconditionally to all transformers versions

  • [GRPO/RLOO] Unify tokenization across all generation backends in _generate_single_turn #5239

Solution

• Gate the workaround with Version("5.3.0") <= Version(transformers.__version__) < Version("5.4.0")
• Pass padding=True only when the workaround is active; omit the argument entirely otherwise (see Warning log: Kwargs passed to processor.__call__ have to be in processor_kwargs dict, not in **kwargs #5502)
• Skip the unpadding step when the workaround is not active, using tokenized["input_ids"] directly
• Correct the comment to reference the right version and link the fix PR

Changes

Bug workaround and version handling:

• Added a conditional check for the transformers library version (>=5.3.0 and <5.4.0) to determine if the padding workaround should be applied, instead of always applying padding. (GRPO, RLOO, and experimental DPPO)
• Updated comments to clarify that the bug is present in transformers 5.3.0 and fixed in 5.4.0, with references to the relevant GitHub issues and PRs.

Prompt ID extraction logic:

• Only unpads input_ids using the attention_mask when the workaround is needed; otherwise, uses the tokenized input_ids directly, simplifying the code path for unaffected versions.

---

Note

Medium Risk
Touches prompt tokenization in multiple trainers; version-gated padding/unpadding could change input IDs for some transformers versions and affect multimodal generation edge cases.

Overview
Restricts the VLM apply_chat_template padding workaround to transformers versions >=5.3.0 and <5.4.0, instead of always forcing padding=True.

When the workaround is inactive, the trainers now skip the unpadding step and use tokenized["input_ids"] directly; comments were updated to reference the correct upstream bug/fix (transformers#44514/#44563).

^{Reviewed by Cursor Bugbot for commit cf3dbe2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

So we were wrong, it was introduced in 5.3 not 5.2?

[albertvillanova]

Yes, I tested locally.

The original comment also incorrectly attributed the bug to transformers 5.2.0