Add Gemma 3 For Sequence Classification

[AhmedHashish123]

Feature request

Hello, I was wondering when will you add support for Gemma3Config for sequence generation as currently only GemmaConfig and Gemma2Config are supported

Motivation

This would be extremely beneficial given that Gemma 2 2B Instruct excels as a sequence classifier. I would expect Gemma 3 4B to be even more performant.

Your contribution

I already did something to finetune Gemma 3 1B by using Gemma2ForSequenceClassification:
In the "modeling_gemma2.py", I import the Gemma 3 text model class as follows:
from ..gemma3.modeling_gemma3 import Gemma3TextModel

and then in "Gemma2ForSequenceClassification" class in the same file, I change this line:
self.model = Gemma2Model(config)
to
self.model = Gemma3TextModel(config)

[zucchini-nlp]

@AhmedHashish123

We usually add a class for task when there is an official checkpoint for the task or when we many requets from community.

In case of gemma3, we can add the Sequence Classification since we've had a lot of request to add it for Vision LLMs with other models (Qwen2-VL and LLaVa). And currently none of the vision LLMs have a classification class. I am totally fine with the addition, and will be happy to review a PR :)

cc @ArthurZucker just in case you don't want the class to be added, unless we get more requests for Gemma3 specifically?

[AhmedHashish123]

Thanks @zucchini-nlp !
I was mainly looking to add it for the text part of the models. Just like the regular LLMs' classifiers where we just remove the last layer and add a linear layer with the output being the desired number of classes.

[zucchini-nlp]

Ah oke. As I noted, we try to not add new classes unless there is a clear need or high community request, so I suggest to wait a bit for this issue to get community reaction. If there is high request, we can add support in the core library :)

[mlukewizard]

Would love this

[saum7800]

would also love to see this!

[SinclairSchneider]

I came across this issue after searching for "sequence classification" and "Gemma 3", so one more person is interested in this feature ;)
Especially since Gemma is exceptionally good for its size, and most models, like BERT or DeBERTa, have a limited sequence length.

[wellinxu]

Useful feature, look forward this.

[zucchini-nlp]

Hey folks, given the high interest for this task, we can add it in core library.

The model will be mostly same as Gemma2 Sequence Classification, with the only difference that it will accept pixel values as input. That means we need a base model for multimodal Gemma3, which I am adding here (#37033). After that we can add Gemma3ForSequenceClassification

[jacob-danner]

Hi, @zucchini-nlp, I'm really interested in using Gemma 3 for sequence classification myself, so I appreciate you moving forward with this!

I haven't contributed to open source before, but it is a goal of mine to start. Is there anything I could do to help with this? I'd be happy to assist if pointed in the right direction

[zucchini-nlp]

@jacob-danner I would prefer the model to inherit from base Gemma3Model class, so we don't duplicate the same code for each possible task. It is annoying to read the code otherwise, let's wait until the base model is added and then I'll ping you here :)

Basically it will be same as Gemma2, with difference that the base model will accept images as well

[DavidAdamczyk]

Hey guys, I'm interested in supporting QA for Gemma 3: #36977. What do you think about this?

[pySilver]

That would be wonderful thing to have Gemma3ForSequenceClassification supported with image support. Gemma 3 is a very very promising set of models.

[kristaller486]

Is there any news on this topic?