Add filter for model type

[ayush1298]

closes #3051
closes #1841

model_type is added in model_meta in PR #3751

[Copilot]

Pull request overview

This PR adds functionality to filter models by their type (dense, cross-encoder, late-interaction) in the MTEB leaderboard and related APIs. The changes introduce a new model_types parameter across multiple files to enable filtering.

Key Changes:

• Added model_types parameter to get_model_metas() function for model filtering
• Added model_types parameter to filter_tasks() function for task filtering
• Integrated a UI checkbox group in the leaderboard app to select model types
• Updated all relevant event handlers to propagate the model_types filter

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
mteb/models/get_model_meta.py	Added model_types parameter to get_model_metas() with filtering logic based on model_meta.model_type field
mteb/leaderboard/app.py	Added UI checkbox for model type selection and threaded the parameter through all filtering functions and event handlers
mteb/filter_tasks.py	Added model_types parameter to filter_tasks() with filtering logic based on metadata.model_types field

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[KennethEnevoldsen]

You have added mo

[ayush1298]

Updated Leaderboard:

When applying filter:

When Cross-encoder is selected in MTEB(Multilingual, v2) benchmark

When late-interaction is selected in ViDoRe(v1, v2) benchmark

Should we also add a column of model_type in the Summary table?

Also, should we add some kind of statistics, like no of models once some model filter is applied(though one can see it directly from the summary table)?

[Samoed]

Should we also add a column of model_type in the Summary table?

I don't think so, because there are too many columns in leaderboard.

Also, should we add some kind of statistics, like no of models once some model filter is applied(though one can see it directly from the summary table)?

I don't think that we need this too

[ayush1298]

Not related to this PR:
While running leaderborad using make run-leaderboard :
I am getting lots of warnings, should we remove some of them. I have listed below some of them which I feel not much essential:

1. https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/model_meta.py#L514:L516
2. Missing subsets {'ru'} for split test, logging like these just filled up terminal, can we just combine all missing subsets/splits for each task and show it once, will be less cluttered.
3. https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/get_model_meta.py#L135-L136
   Couldn't get why its try to fetch these models, I think these are the models for which we dont have implementation in MTEB. Logs are like:

Could not get source model: "Model 'Qwen/Qwen3-0.6B' not found in MTEB registry. Did you mean: 'Qwen/Qwen3-Embedding-0.6B' or Qwen/Qwen3-Embedding-8B?" in MTEB

Could not get source model: "Model 'NeuML/pubmedbert-base-embeddings' not found in MTEB registry. Did you mean: 'NeuML/pubmedbert-base-embeddings-8M' or NeuML/pubmedbert-base-embeddings-2M?" in MTEB

[ayush1298]

Should we also add a column of model_type in the Summary table?

I don't think so, because there are too many columns in leaderboard.

Anyway, we have a scrollbar there. These is just, so that one can easily know type of model there only

[KennethEnevoldsen]

@ayush1298 I would make a seperate issue on the warnings (but would be nice to clean them up a bit)

[ayush1298]

I have added tests

[KennethEnevoldsen]

this does not test the get_model_meta

[ayush1298]

Sorry, yes, it would just check filled in ModelMeta. Should I change it to below?

@pytest.mark.parametrize("model_type", ["dense", "cross-encoder", "late-interaction"])
def test_get_model_metas_each_model_type(model_type):
    models = mteb.get_model_metas(model_types=[model_type])
    
    for model in models:
        assert model_type in model.model_type

[KennethEnevoldsen]

Yes

[ayush1298]

@KennethEnevoldsen I have added tests

[Samoed]

Can you create leaderboard that we can test?

[ayush1298]

Can you create leaderboard that we can test?

I have forgotten this one. It was 1st to duplicate hf space, but then how I should run it from these branch?

[Samoed]

You can set it in dockerfile of your space

[ayush1298]

The link to hf space:
https://huggingface.co/spaces/AyushM6/leaderboard_filter_model_type

[Samoed]

I think if nothing is selected then there shouldn't be models

[Samoed]

But overall this is working

[ayush1298]

I think if nothing is selected, then there shouldn't be models

Not getting why the filter is not working correctly here. I don't think the filter is wrong here, but somehow not able to find why it's happening.
@Samoed Could you please check it?

[Samoed]

Yes, I'll check later

[KennethEnevoldsen]

I think if nothing is selected, then there shouldn't be models

I think it is fair to assume that if everything is unselected the filter is not applied (I would leave it as is)

[Samoed]

I think it is fair to assume that if everything is unselected the filter is not applied (I would leave it as is)

I'm not sure. I fixed this in last commit eb75377

[KennethEnevoldsen]

I'm not sure. I fixed this in last commit eb75377

So default is all boxes checked?

[Samoed]

Yes

[KennethEnevoldsen]

I don't have a strong opinion here, but we should be consistent with the other selectors. I see that we are currently inconsistent in our radio buttons:

This is probably a separate issue, though.

[ayush1298]

Can we merge this one? @Samoed