Resolve PLaMo2 issue with transformers>=4.53

[Alnusjaponica]

Purpose

Resolve #22999

Test Plan

Test Result

(Optional) Documentation Update

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[hmellor]

Hi @Alnusjaponica, does the plamo2 test pass for you locally?

[Alnusjaponica]

I’ll close this PR as this change is included by #23998

[Alnusjaponica]

@hmellor
Apologies for the delayed reply.
The unit tests currently depend on mamba-ssm and causal-conv1d, which is why this PR has remained a draft. #23998 primarily targets V1 engine support but partially relies on the changes here, so we’ve included the bugfix in this PR. We’d like to re-enable the unit tests as well, but it likely makes more sense to align on V1 support first and then restore the tests.
#23520 (comment)