[v2] fix contriever (add similarity_fn_name to ST wrapper)

[Samoed]

Closes #1731

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[Samoed]

@isaac-chung Model loading gives

System.IO.IOException: No space left on device : '/home/runner/runners/2.321.0/_diag/Worker_20250110-152956-utc.log' ...

[sam-hey]

Just as an idea: I believe the ModelMeta object already has this information similarity_fn_name="cosine". Why not add an Enum type to specify the evaluation function? This would make it much more straightforward to add new models. There's no need to implement a new wrapper class—just one Enum pointing to the fn

class EvaluationFunction(Enum):
    DOT_PRODUCT = dot_product
    MAX_SIM = max_similarity
    COSINE = cosine_similarity

[Samoed]

Yes, I wanted to integrate ModelMeta with the wrapper somehow, but for now, they’re independent. I’m not sure if it’s a good idea to pass arguments from ModelMeta to the loader. Thank you for ideas!

[sam-hey]

Please take a look at the screenshot. The MTEB ModelMeta is already available.

https://github.com/embeddings-benchmark/mteb/blob/v2.0.0/mteb/evaluation/evaluators/model_classes.py#L324:L326

[KennethEnevoldsen]

Isn't it possible to have the model self define their similarity function? Any reason for us to create a custom wrapper?

[KennethEnevoldsen]

(if this is a one-of I would just move it next to the special case)

[Samoed]

Model that trained with sentence transformers have ability to provide it, but this model doesn't have config for this as was discussed in #1731. I think there can be more models with different similarity functions. Maybe we can use approach from #1759

[isaac-chung]

Have we tried defining a Model Meta for the contriever model with the dot similarity function? I thought that was the same way we implemented Colbert with @sam-hey but with max-sim.

[Samoed]

For colbert models created different wrapper with max-sim

[sam-hey]

          > it's a bit surprising that ModelMeta.similarity_fn_name isn't being utilized.

We would love to switch to that one and would encourage a PR for this. It was only recently added to haven't been integrated throughout.

Originally posted by @KennethEnevoldsen in #1592 (comment)

---

I believe ModelMeta was never fully utilized. Instead of reverting to the previous version—which I agree was a better approach—we might be able to resolve these issues by properly implementing ModelMeta now.

@isaac-chung's comment :

I'd like to understand why, and perhaps convince us to move/revert back to what we had before, as to me, it worked better than needing to define separate wrappers per scoring function, which isn't scalable.

[Samoed]

I've updated the approach so that the similarity function can now be directly passed to the wrapper.

Previously, users had to specify which similarity function to use, which was a hidden feature and independent of the models.

evaluation.run(
    model,
    score_function="max_sim",
)

[Samoed]

I’ve updated it again, and now it uses a string similarity_fn_name like it stated in ModelMeta. I suggest leaving it as is for now and integrate ModelMeta parameters in a separate PR, as we’re currently duplicating model names and revisions for every model.

[isaac-chung]

Thanks all. I felt that we could have improved docs for the previous 'hidden' feature, rather than removing it.

I agree that we can improve the ModelMeta param integration in a separate PR.

[Samoed]

The previous implementation required users to explicitly pass the scoring function, which could lead to issues with results that were hard to reproduce or match. This new approach reduces user input.

[isaac-chung]

Thanks for iterating! Feel free to skip the model loading test while I work on a fix.

Perhaps we could repurpose #1759 a bit now for the model meta integration.