Models missing after leaderboard code refresh

[sfc-gh-lmerrick]

For some reason the new leaderboard only displays Snowflake's Arctic Embed v1.5 model, dropping both our v1 and v2 models from the listings. The actual MTEB eval results are still present in all v1 and v2 models' model cards, so it seems that something went awry in the leaderboard code changes themselves.

[Samoed]

Can you please provide more information about the leaderboard you are referring to? I am able to see all the models on the MTEB (Multilingual, v1) leaderboard.

[sfc-gh-lmerrick]

Apologies for the delayed response, I didn't see your reply until today!

MTEB(eng, v1)

The original English benchmark by Muennighoff et al., (2023). This page is an adaptation of the old MTEB leaderboard.

This page is missing our models, despite them showing up on the old MTEB leaderboard.

Results for our models also appear to be missing from the BEIR page, despite MTEB Retrieval being adapted from BEIR and the datasets overlapping (as seen in the figure from the MTEB paper)

[KennethEnevoldsen]

They do appear if you press "Allow All":

They are simply being filtered now due to not being zero-shot on the leaderboard.

As an important provider in the embedding models space we would love to hear your opinion on how we should set the default over on #2119 (which is currently ongoing, but we will probably make a decision soon-ish)

[KennethEnevoldsen]

I believe this resolved this issue, but do add to the discussion over on #2119

[sfc-gh-lmerrick]

Ah, I think I have found the source of the confusion:

MTEB(eng, v1)

The original English benchmark by Muennighoff et al., (2023). This page is an adaptation of the old MTEB leaderboard.

This description is misleading, because the page has changed which models are displayed. I will chime in separately on #2119, but the choice of showing all models which do not report their training data but disqualifying those which do report it gives a very misleading view!