Split VisualSTS into eng and multi

[isaac-chung]

To create the VisualSTS(eng) and VisualSTS(multi) columns in the LB for the MIEB benchmarks (to align with the paper), we split the tasks by using aggregate tasks, so that we don't need to rerun results again, and only need to generate new ones.

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the results files checker make pre-push.

Adding a model checklist

• I have added model implementation to mteb/models/ directory. Instruction to add a model can be found here in the following PR: feat: Add MIEB and MIEB-lite as benchmarks mteb#2035

[KennethEnevoldsen]

hmm, what a hacky way of using the aggregate task ;)

Like it though. We could also do this for the BRIGHT tasks (on the old leaderboard the split appears as different tasks)

Should we by default exclude aggregate tasks for mteb.get_tasks()?

[isaac-chung]

hmm, what a hacky way of using the aggregate task ;)

Like it though. We could also do this for the BRIGHT tasks (on the old leaderboard the split appears as different tasks)

Should we by default exclude aggregate tasks for mteb.get_tasks()?

Thanks!

And I don't have a strong preference. I guess if they are excluded by default, a parameter would added to "enable" them? e.g. include_aggregate=True

[KennethEnevoldsen]

And I don't have a strong preference. I guess if they are excluded by default, a parameter would added to "enable" them? e.g. include_aggregate=True

hmm yea. Let us just include them for now, but could imagine that we could include a lot of "duplicates".

[gowitheflow-1998]

looks great. Scores are matching with the aggregated results in the paper too. Do we have all models here?

[isaac-chung]

looks great. Scores are matching with the aggregated results in the paper too. Do we have all models here?

@gowitheflow-1998 only missing blip2 models. Could I get some help running them please? [update] All run.