Maeb merge main v2#3447

Merged

Samoed merged 217 commits intomaebfrom

maeb_merge_main_v2

Oct 22, 2025

Member

Samoed commented Oct 20, 2025 •

edited

Loading

I've make all tasks and files to follow pep8
Made torchadio as optional dependency
Removed duplicates from v1

makram93 and others added 30 commits

July 11, 2025 22:06


          model: add image support for jina embeddings v4 (#2893)

17be7e5

* feat: unify text and image embeddings for all tasks

* fix: uniform batch size

* fix: update error message

* fix: update code task

* fix: update max length

* fix: apply review suggestions


          model: add kalm_models (kalm-emb-v2) ModelMeta (new PR) (#2889)

9ecac21

* feat: add KaLM_Embedding_X_0605 in kalm_models

* Update kalm_models.py for lint format

* kalm-emb-v2

* kalm-emb-v2

* kalm-emb-v2

* kalm-emb-v2

* kalm-emb-v2

---------

Co-authored-by: xinshuohu <[email protected]>
Co-authored-by: Xinshuo Hu <[email protected]>


          Add Classification Evaluator unit test (#2838)

4a47f90

* Adding Classification Evaluator test

* Modifications due to the comments

* Update tests/test_evaluators/test_ClassificationEvaluator.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* Update tests/test_evaluators/test_ClassificationEvaluator.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* Modifications due to the comments

* Modifications due to the comments

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>


          fix: update colpali engine models (#2905)

9864e2a

* adding vidore benchmarks

* fix typo

* clean vidore names + per lang eval

* lint

* vidore names

* bibtex fix

* fix revision

* vidore v2 citation

* update citation format and fix per-language mappings

* lint: citations

* typo citations

* fix revisiions

* lint

* fix colnomic3b revision

* fix colqwen2.5 revision + latest repo version

* fix query agmentation tokens

* colsmol revision


          1.38.35

5a8ccec

Automatically generated by python-semantic-release


          Evaluator tests (#2910)

c7078af

* Adding Classification Evaluator test

* Modifications due to the comments

* Update tests/test_evaluators/test_ClassificationEvaluator.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* Update tests/test_evaluators/test_ClassificationEvaluator.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* Modifications due to the comments

* Modifications due to the comments

* Adding STSEvaluator and SummarizationEvaluator tests

* Correcting due to the comments

* Correcting due to the comments

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>


          Classification dataset cleaning (#2900)

aef1e33

* Classification dataset cleaning

* Update pull request number

* Fix metadata test

* fix formatting

* add script for cleaning


          Update tasks & benchmarks tables

56c98ed


          dataset: Add JapaneseSentimentClassification (#2913)

57438c2

Add JapaneseSentimentClassification


          Update tasks & benchmarks tables

372fc4c


          fix: change passage prompt to document (#2912)

a298fa9

* change document to passage

* fix prompt names

* fix kwargs check

* fix default prompt


          1.38.36

8eb4f6d

Automatically generated by python-semantic-release


          model: Add OpenSearch inf-free sparse encoding models (#2903)

5a868e3

add opensearch inf-free models

Co-authored-by: Isaac Chung <[email protected]>


          dataset: add BarExamQA dataset (#2916)

1dcc6dc

* Add BareExamQA retrieval task

* ran linter

* updated details

* updated details

* fixed subtype name

* fixed changes

* ran linter again


          Use mteb.get_model in adding_a_dataset.md (#2922)

c1922c8

Update adding_a_dataset.md


          fix: specify revision for opensearch (#2919)

0ac0231

specify revision for opensearch


          1.38.37

b12b926

Automatically generated by python-semantic-release


          Update the link for gemini-embedding-001 (#2928)

533ce59


          fix: replace with passage (#2934)

5ed6c90


          fix: Only import SparseEncoder once sentence-transformer version have…

79a43af

… been checked (#2940)

* fix: Only import SparseEncoder once sentence-transformer version have been checked

fixes #2936

* Update mteb/models/opensearch_neural_sparse_models.py

Co-authored-by: Isaac Chung <[email protected]>

---------

Co-authored-by: Isaac Chung <[email protected]>


          fix: Prevent incorrectly passing "selector_state" to get_benchmark (#…

8496ec2

…2939)

The leaderboard would have (silent) errors where `get_benchmark` lead to a KeyError due to "selector_state" being passed as a default value. Setting `DEFAULT_BENCMARK_NAME` as the value solves this issue.


          docs: Update adding_a_dataset.md (#2947)

a78debf

* docs: Update adding_a_dataset.md

* Update docs/adding_a_dataset.md


          ci: bump semantic release

4ef8571


          1.38.38

03a0582

Automatically generated by python-semantic-release


          dataset: Add BSARD v2, fixing the data loading issues of v1 (#2935)

* BSARD loader fixed

* BSARDv2 metadata fixed

* Update mteb/tasks/Retrieval/fra/BSARDRetrieval.py

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>


          Update tasks & benchmarks tables

da46c8e


          dataset: add GovReport dataset (#2953)

42dfe0d

* Added govreport task

* Updated description


          dataset: add BillSum datasets (#2943)

007d19f

* Added BillSum datasets

* fixed billsumca

* Updated BillSumCA description

* Updated BillSumUS description

* Update mteb/tasks/Retrieval/eng/BillSumCA.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* Update mteb/tasks/Retrieval/eng/BillSumUS.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

* lint

* lint

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Isaac Chung <[email protected]>


          Update tasks & benchmarks tables

e4f30e9


          fix: Add new benchmark beRuSciBench along with AbsTaskTextRegression (#…

36df9ca

…2716)

* Add RuSciBench

* fix bitext mining lang

* Add regression task

* fix init

* add missing files

* Improve description

* Add superseded_by

* fix lint

* Update regression task to match with v2

* Add stratified_subsampling for regression task

* Add boostrap for regression task

* Rename task class, add model as evaluator argument

* fix import

* fix import 2

* fixes

* fix

* Rename regression model protocol

semantic-release and others added 4 commits

October 20, 2025 12:31


          2.0.0

51412db

Automatically generated by python-semantic-release


          bump version

838f25f


          Merge branch 'main' of https://github.com/embeddings-benchmark/mteb

48b11a7


          Merge branch 'main' into maeb_merge_main_v2

493ce4f

Contributor

KennethEnevoldsen commented Oct 20, 2025

Do you want to do all of this in one go? I would probably just transfer one task type over at a time

Member Author

Samoed commented Oct 20, 2025

I just want to make basic merge firstly to make tests runnable. After this, I will update per task type

Samoed changed the base branch from maeb_v2 to maeb

October 20, 2025 19:14

Samoed added 11 commits

October 20, 2025 23:46


          linter pass

b457921


          make mteb importable

6b8f98d


          add audio to test dependencies

3f52798


          remove metadata_dict

cf871d3


          fix tests

ac9c19e


          fix tests

81d0c96


          make torchaudio optional

5fac1f5


          fix retrieval init

632673c


          fix imports

0b23206


          fix retrieval task and torch audio

050107d


          move more torcaudio imports

e060767

Samoed marked this pull request as ready for review

October 21, 2025 11:05

Samoed requested a review from KennethEnevoldsen

October 21, 2025 11:07

Samoed assigned AdnanElAssadi56 and unassigned AdnanElAssadi56

Samoed requested a review from AdnanElAssadi56

October 21, 2025 11:08

KennethEnevoldsen approved these changes

View reviewed changes

Contributor

KennethEnevoldsen left a comment

This is impossible to review. Gotta trust you here

Member Author

Samoed commented Oct 21, 2025

You can view last 11 commits

Contributor

KennethEnevoldsen commented Oct 22, 2025

Ahh yea that was a good idea. Yeah changes looks reasonable!

Samoed merged commit 9f1c7a6 into maeb

10 checks passed

Samoed deleted the maeb_merge_main_v2 branch

October 22, 2025 11:46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet