update results for giga-embeddings-instruct

[ekolodin]

Checklist

• My model has a model sheet, report or similar
• My model has a reference implementation in mteb/models/ this can be as an API. Instruction on how to add a model can be found here
  • No, but there is an existing PR ___
• The results submitted is obtained using the reference implementation
• My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
• I solemnly swear that for all results submitted I have not on the evaluation dataset including training splits. If I have I have disclosed it clearly.

[Samoed]

You need to put your new results in dir witn model revision

[ekolodin]

You need to put your new results in dir witn model revision

In what dir?

[Samoed]

In your case you should put in /results/ai-sage__Giga-Embeddings-instruct/646f5ff3587e74a18141c8d6b60d1cffd5897b92/

[ekolodin]

where did you get this string 646f5ff3587e74a18141c8d6b60d1cffd5897b92?

[Samoed]

Revision of your model when I added it
https://huggingface.co/ai-sage/Giga-Embeddings-instruct/commit/646f5ff3587e74a18141c8d6b60d1cffd5897b92

https://github.com/embeddings-benchmark/mteb/blob/d5ccf108686254d98c3589b06deaeb3e186ac1b5/mteb/models/ru_sentence_models.py#L606

[ekolodin]

but i changed the model weights, should this revision id be changed?

[ekolodin]

ok, i've moved to revision_id dir

[Samoed]

Yes, you should change revision of your model in main repo too

[ekolodin]

So, I've updated the revision_id on the latest one

[Samoed]

You should update revision in main repo too, and after that run your model with mteb implementation @ekolodin

[ekolodin]

what main repo? this is not main repo? i do not understand you @Samoed

[Samoed]

I mean to https://github.com/embeddings-benchmark/mteb

[ekolodin]

I've made it @Samoed

embeddings-benchmark/mteb#2774

[ekolodin]

Any updates?

[Samoed]

No, you need to run your model with implementation from mteb.

[ekolodin]

I've already run it, and the push the results.

[ekolodin]

@Samoed could you please merge?

[Samoed]

As I said I don't think you've correctly run your model. I've got

Task	MTEB results	Your PR
TERRa	0.638273	0.583729
CEDRClassification	0.6247	0.684857

Please, rerun your model with latest mteb.

https://www.kaggle.com/code/samoed/notebook908a227956/notebook?scriptVersionId=244761810

[ekolodin]

I think you use wrong instructions for benchmarks

[ekolodin]

Where do you set up instructions in your example? I can't see it, we have custom instructions for each task

[Samoed]

we have custom instructions for each task

If you have custom instructions, then you should add them for your model. We're integrating KALM that using custom instructions for tasks embeddings-benchmark/mteb#2478, too. I'll try to merge this PR soon

[ekolodin]

Where can I get your default instructions?

[Samoed]

They are defined in task Metadata, for example for TERRa or if instruction is not defined in task then default instruction for task type will be used, e.g. PairClassification. Instructions are selected by this function https://github.com/embeddings-benchmark/mteb/blob/e6238f2305be79f7b32d934d85ef4557fb89cb22/mteb/models/wrapper.py#L91

[ekolodin]

I have added custom instructions code: embeddings-benchmark/mteb#2836

[Samoed]

Now I verify scores