[v2] Remove multilingual task

[Samoed]

Closes #1832

I've removed MultilingualTask and explicitly added is_multilingual to tasks, which was previously handled by MultilingualTask. I also moved the fast_loading method from MultilingualTask to AbsTask, but I think we should reupload these datasets in the correct format.

Additionally, we might want to determine is_multilingual automatically based on eval_langs, but I’m not entirely in favor of this since it feels a bit like magic. Another option is to integrate is_multilingual into TaskMetadata.

Code Quality

• Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

• Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

• New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
• Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

[KennethEnevoldsen]

I have also made some changes on this over on main. I would probably do a merge before (my fix removed .is_multilingual). It was required to allow for the aggregate task I believe.

[Samoed]

No, is_multilingual is still in AbsTask.

mteb/mteb/abstasks/AbsTask.py

Line 65 in dba7a95

is_multilingual: bool = False

[KennethEnevoldsen]

(we could also consider merging this and resolving it in a merge/partial merge)

[KennethEnevoldsen]

No, is_multilingual is still in AbsTask.

Yes, but it is not in .evaluate (which I see it also isn't here, that is my mistake).

So yea, no merge is needed

[KennethEnevoldsen]

Alright, looking a bit more into this. Are there cases where we can't replace .is_multilingual?

[Samoed]

No, we can totally remove is_multilingual. If you think we should do this, I will remove it

[isaac-chung]

Maybe I'm missing something: how do we determine which subsets to load? Does is_multilingual play a role or do we just check for the presence of a default subset?

[Samoed]

how do we determine which subsets to load?

By hf_subsets. If user want to run some part of them, then they require to run mteb.get_tasks that will filter required subsets. For example, for language filter.

mteb/mteb/abstasks/AbsTask.py

Line 315 in dba7a95

self.hf_subsets = subsets_to_keep

Does is_multilingual play a role or do we just check for the presence of a default subset?

Previously yes, but now we can get subsets with hf_subsets_to_langscripts (maybe we can create separate method just for getting splits), that return them correctly. For now, I think we can totally remove it

[isaac-chung]

Sorry, I didn't phrase the question right. What I'm curious about is the evidence supporting this statement:

we can totally remove is_multilingual.

[Samoed]

Sorry, I didn't phrase the question right. What I'm curious about is the evidence supporting this statement:

we can totally remove is_multilingual.

Yes, for now, is_multilingual is only used to check the default split.

hf_subsets = list(self.dataset) if self.is_multilingual else ["default"]

[isaac-chung]

Yes, for now, is_multilingual is only used to check the default split.

Hmm it seems like there are a few more references, like

• In descriptive stats:

  mteb/mteb/abstasks/AbsTask.py

  Line 290 in d0eb782

  if self.is_multilingual:

• In upload dataset to hub:

  mteb/mteb/abstasks/AbsTask.py

  Line 386 in d0eb782

  def _upload_dataset_to_hub(self, repo_name: str, fields: list[str]) -> None:

Do they seem easy to refactor?

[Samoed]

I think we can make is_multilingual just property of TaskMetadata that would be calculated based on eval_splits for using in some parts of mteb

[isaac-chung]

Looks good! Just got one small suggestion.