init automate script

[Samoed]

Closes https://github.com/embeddings-benchmark/results/issues/196

I've created CI action that would create tables for PR automatically. If there won't be any results for multilingual-e5 or gemini-embedding-001 then results of only new model will be added.

With each new commit CI will update comment with results.

Checklist

• My model has a model sheet, report or similar
• My model has a reference implementation in mteb/models/ this can be as an API. Instruction on how to add a model can be found here
  • No, but there is an existing PR ___
• The results submitted is obtained using the reference implementation
• My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
• I solemnly swear that for all results submitted I have not on the evaluation dataset including training splits. If I have I have disclosed it clearly.

[github-actions]

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: ai-forever/FRIDA
Tasks: MassiveIntentClassification

Results for ai-forever/FRIDA

task_name	ai-forever/FRIDA	google/gemini-embedding-001	intfloat/multilingual-e5-large	Max result
MassiveIntentClassification	0.79	0.82	0.6	0.85
Average	0.79	0.82	0.6	0.85

---

[KennethEnevoldsen]

Looks good.

A few things which we could add, but is not required to merge this (probably out of scope):

• Would love to also include the "max" score the each task
• Would love to be able trigger this command using something like @bot result --models intfloat/e5-large-v2

[Samoed]

Would love to be able trigger this command using something like @bot result --models intfloat/e5-large-v2

To use it like a bot, we need to host it somewhere. I can host it, but I'm not sure if that's the best option. We could make it trigger on each comment and check if there’s a command to run, but I’m not sure if that’s a good idea.

Another solution could be to parametrize workflow dispatch (manually trigger) to add models here. I will try this approach

[Samoed]

I've added support for max score and comparison with different models through workflow_dispatch, but I can't test it yet because it hasn't been merged

[Samoed]

@KennethEnevoldsen Can you review this PR? Actual work of the script in this comment #220 (comment)

[KennethEnevoldsen]

Looks good! Feel free to merge

To use it like a bot, we need to host it somewhere. I can host it, but I'm not sure if that's the best option. We could make it trigger on each comment and check if there’s a command to run, but I’m not sure if that’s a good idea.

Another solution could be to parametrize workflow dispatch (manually trigger) to add models here. I will try this approach

I think for now it is perfectly fine to just have it as a default comment, and then we can run any additional stuff ourselves.