Add results for KaLM-Team/KaLM_Embedding-X-0605 in MMTEB

[YanshekWoo]

Checklist

• My model has a model sheet, report or similar: KaLM-Embedding
• My model has a reference implementation in mteb/models/ this can be as an API. Instruction on how to add a model can be found here
  • No, but there is an existing PR: add kalm_models ModelMeta
• The results submitted is obtained using the reference implementation
• My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
• I solemnly swear that for all results submitted I have not on the evaluation dataset including training splits. If I have I have disclosed it clearly.

[KennethEnevoldsen]

This is dependent on the PR, where an implementation is missing, which @Samoed will add. Once added, we should probably test that the score aligns so I will assign this to @Samoed. @Samoed feel free to request a review at the end if you need a second pair of eyes on it

[YanshekWoo]

@Samoed
Excuse me, I'd like to know the progress of this Merge Request.
Are there any difficulties or areas where you need my clarification or assistance?

[Samoed]

We need to integrate your model firstly. Can you review metric results from our current implementation embeddings-benchmark/mteb#2478 (comment)
@YanshekWoo

[YanshekWoo]

Before merging the results, can we update them first?

By the way, could we raise an issue here regarding the evaluation instructions for MTEB?

[Samoed]

By the way, could we raise an issue here regarding the evaluation instructions for MTEB?

I'm not sure what you want. Can you update your implementation in embeddings-benchmark/mteb#2775 using new loader for Kalm models from embeddings-benchmark/mteb#2478?

[YanshekWoo]

I'm not sure what you want. Can you update your implementation in embeddings-benchmark/mteb#2775 using new loader for Kalm models from embeddings-benchmark/mteb#2478?

The Instruction we evaluated has been updated, so the results for each dataset will also change accordingly.

We used SentenceTransformer for loading and specifying prompts during our evaluation, but we found that there is an issue with the current mteb code when specifying specific task instructions:
if the task dataset contains a "-", there will be an error in parsing the task_name, such as when specifying "NQ-PL-query".

https://github.com/embeddings-benchmark/mteb/blob/d7ff1ab3168ddf765d497b22e928af355f0cffe6/mteb/models/wrapper.py#L77

[Samoed]

The Instruction we evaluated has been updated, so the results for each dataset will also change accordingly.

Can you update them in your implementation?

if the task dataset contains a "-", there will be an error in parsing the task_name, such as when specifying "NQ-PL-query".

I see. I will fix it

[YanshekWoo]

Can you update them in your implementation?

Yes. I will update it today.

[YanshekWoo]

Can you update them in your implementation?

I have created a new PR of the model card at: embeddings-benchmark/mteb#2850

And the new results with new instruction are updated in new PR: #227