Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model

[albertvillanova]

Fix UNEXPECTED lm_head.weight warning when loading a CausalLM as a reward model.

This PR refines how warnings are suppressed when loading a Causal Language Model (CausalLM) checkpoint into an AutoModelForSequenceClassification. The changes ensure that both expected "missing" and "unexpected" key warnings are suppressed, improving clarity for users and maintaining compatibility across different transformers library versions.

Problem

Loading a CausalLM checkpoint into AutoModelForSequenceClassification (as RewardTrainer does when model is a string) produces two harmless LOAD REPORT warnings since transformers v4.57.2:

• MISSING score.weight — new seq-clf head, not in the checkpoint, randomly initialized.
• UNEXPECTED lm_head.weight — causal LM head, in the checkpoint but absent from seq-clf.

suppress_seqcls_warning already suppressed the MISSING side. See:

• Fix logging warning suppression with scoped override for seq-clf head key #5058
• Fix logging warning suppression for transformers 4.56.2 #5077

However, the UNEXPECTED warning is not suppressed yet: https://github.com/huggingface/trl/actions/runs/23131343032/job/67185204370

Qwen2ForSequenceClassification LOAD REPORT from: trl-internal-testing/tiny-Qwen2ForCausalLM-2.5
Key            | Status     |  | 
---------------+------------+--+-
lm_head.weight | UNEXPECTED |  | 

Notes:
- UNEXPECTED	:can be ignored when loading from different task/architecture; not ok if you expect identical arch.

Solution

This PR extends it to also suppress the UNEXPECTED side, and renames the internal helper to reflect that it now covers both directions.

Changes

Warning suppression improvements:

• Updated the warning suppression logic to handle both "missing" (score.weight) and "unexpected" (lm_head.weight) keys when loading a CausalLM checkpoint into a sequence classification model, providing clearer comments and more robust regex patterns.
• Enhanced the context manager for newer transformers versions (>= 4.57.0) to ignore both missing and unexpected keys by updating _keys_to_ignore_on_load_missing and introducing _keys_to_ignore_on_load_unexpected on GenericForSequenceClassification.
• Updated the version-aware suppression wrapper to use the new context manager for recent transformers versions and the improved logging filter for older versions, ensuring consistent behavior.

---

Note

Low Risk
Low risk: only adjusts warning-suppression logic during reward-model loading to also ignore expected lm_head.* unexpected keys on newer transformers versions.

Overview
When loading a CausalLM checkpoint via AutoModelForSequenceClassification in RewardTrainer, the PR expands warning suppression to cover both expected cross-architecture load reports: missing score.weight and (in transformers>=4.57.0) unexpected lm_head.* keys.

It renames/clarifies the internal context managers and, for newer transformers, temporarily sets both GenericForSequenceClassification._keys_to_ignore_on_load_missing and _keys_to_ignore_on_load_unexpected, while keeping the logging-filter fallback for older versions.

^{Written by Cursor Bugbot for commit a91d7b1. This will update automatically on new commits. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Perfect