Add MambaForSequenceClassification

[mjschock]

What does this PR do?

Adds MambaForSequenceClassification for sequence classification with the MambaModel backbone.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker thanks for your work bringing in Mamba! I'm wondering if there's any objection to adding a MambaForSequenceClassification model? I followed the template example as best as I could and happy to continue with adding the new test and finishing it up.

[amyeroberts]

Hi @mjschook, thanks for opening this PR and adding this!

AFAICT, these changes seem reasonable to me. @ArthurZucker is off for a week - so let's wait for him to come back to confirm if there's any reason for not adding this to Mamba.

A few things that will need to be added:

• Tests for the model i.e. equivalent to create_and_check_mamba_model and the model should be added to all_model_classes
• The model needs to be documented in mamba.md
• All the tests in the CI should be passing

[amyeroberts]

The activation layer should be set in the init and then called here i.e.

    def __init__(...):
        ...
        self.activation = ACT2FN[config.hidden_act]

    def forward(...):
        ...
        x = self.activation(x)

[mjschock]

I'm not sure whether we actually want to freeze the params for the base model here, but I found there's a test specific to sequence classification that expects all the unfrozen params to be initialized in the range [0.0, 1.0] and the initialized values for the mixer don't satisfy that assertion.

So... I froze them and made sure the classification head params were initialized to satisfiy the test. It makes intuitive sense to me to freeze them in the case of transfer learning for this task and I did confirm that running LoRA PEFT with target_modules=["x_proj", "embeddings", "in_proj", "out_proj"] and task_type=TaskType.SEQ_CLS does unfreeze the target modules so it appears to work fine, but not sure if we want to force them to be frozen by default.

Anyway, happy to adjust if there's a better practice to follow here.

[ArthurZucker]

Hey! We usually do the following before merging such PRs:

1. Create a feature request issue to add this new class
2. Wait until the community picks this up
3. Wait until there are actually pretrained checkpoints released by the community or the authors.

As is, this does not really help anyone as it can be easily implemented by anyone that wants to train a model no?

[mjschock]

Thanks for the feedback @ArthurZucker - I'll close it since I went another direction, using prompt tuning instead. I'll keep the process you laid out in mind for the future. =)

[scottfleming]

Would be helpful to have this class. Looking forward to #30431