Docs: update docs according to current state by Samoed · Pull Request #1870 · embeddings-benchmark/mteb

Samoed · 2025-01-24T22:09:01Z

Checklist

Run tests locally to make sure nothing is broken using make test.
Run the formatter to format the code using make lint.

Closing #1868

isaac-chung

Thanks for quickly starting the effort!

Added a few suggestions. We should also update the README file as well since one file is removed. There, we should also add a line pointing out that the "adding a model" instructions is equivalent to adding a model to the leaderboard.

docs/adding_a_benchmark.md

docs/adding_a_model.md

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

isaac-chung · 2025-01-28T14:15:39Z

Thanks. We just need to add this part then we're ready to merge:

We should also update the README file as well since one file is removed. There, we should also add a line pointing out that the "adding a model" instructions is equivalent to adding a model to the leaderboard.

Samoed · 2025-01-28T15:39:45Z

@x-tabdeveloping @KennethEnevoldsen Do you have something to add?

isaac-chung

Looks good to me. Thanks for iterating!

README.md

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

isaac-chung · 2025-01-30T10:49:04Z

Going to merge now. Feel free to add anything extra in a separate PR.

* Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.4 Automatically generated by python-semantic-release * Update tasks table * fix: Limited plotly version to be less than 6.0.0 (#1902) Limited plotly version to be less than 6.0.0 * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * update stella/jasper metainfo (#1896) update stella meta * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.5 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Feat: Add FaMTEB (Farsi/Persian Text Embedding Benchmark) (#1843) * Add Summary Retrieval Task * Add FaMTEBClassification * Add FaMTEBClustering * Add FaMTEBPairClassification * Add FaMTEBRetrieval and BEIRFA and FaMTEBSTS * Add FaMTEBSummaryRetrieval * Add FaMTEB to benchmarks * fix benchmark names * temporary fix metadata * Fix dataset revisions * Update SummaryRetrievalEvaluator.py * Update task files * Update task files * add data domain and subtask description * Update AbsTaskSummaryRetrieval and FaMTEBSummaryRetrieval * Update AbsTaskSummaryRetrieval * Add mock task * Update AbsTaskSummaryRetrieval * Update AbsTaskSummaryRetrieval * make lint * Refactor SummaryRetrieval to subclass BitextMining * Add aggregated datasets --------- Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> * Update tasks table * Docs: update docs according to current state (#1870) * update docs * Apply suggestions from code review Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * update readme * Update README.md Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Adding a banner to the new MMTEB leaderboard (#1908) * Adding a banner to the new MMTEB leaderboard * linting * Update mteb/leaderboard/app.py Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * adding reference to mteb arena --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: Filling missing metadata for leaderboard release (#1895) * Update ArxivClusteringS2S.py * fill some metadat for retrieval * fill in the reste of missing metadata * fix metadata * fix climatefever metadata * fix: Added CQADupstack annotations * removed annotation for non-exisitant task * format * Added financial to other financial dataset * Moved ArguAna annotation to derivate datasets --------- Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.6 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: remove SummaryRetrieval as a type (#1915) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: revert rename and add to description (#1918) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * docs: Add sort to domains for task metadata (#1922) Tests currently go into an infinite loop. This should prevent that. * Update tasks table * 1.31.7 Automatically generated by python-semantic-release * docs: Updated citation for mteb(scandinavian) (#1914) fix: Updated citation for mteb(scandinavian) * fix: Add datasets in CodeRAG-Bench (#1595) * add three out of four datasets in CodeRAG-Bench * add verified CodeRAGStackoverflowPostsRetrieval dataset * clean up code and make some comments * fixed lint errors * addressed comments about code-rag datasets: fixed grammar and remove unnessary code and loop * roll back files which is not supposed to change * fixed the comments in split_by_first_newline() and make the methods private by adding a underscore prefix * refactor to use common args * update task descriptions * add entry in benchmarks * correct the alphanumeric order for the dataset * add in tasks.md * add in tasks.md * update task metadata * update importing path * fix lint errors * correct CodeRAG task metadata description field and id for stackoverflow-posts * fix error in test --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * 1.31.8 Automatically generated by python-semantic-release * Leaderboard: Acks (#1930) Add acs * omit instructions.py --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Márton Kardos <power.up1163@gmail.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Mehran Sarmadi <128898167+mehran-sarmadi@users.noreply.github.com> Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> Co-authored-by: Wissam Siblini <36303760+wissam-sib@users.noreply.github.com> Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> Co-authored-by: Pengfei He <hepengfe@gmail.com> Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>

* Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * update stella/jasper metainfo (#1896) update stella meta * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.5 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Feat: Add FaMTEB (Farsi/Persian Text Embedding Benchmark) (#1843) * Add Summary Retrieval Task * Add FaMTEBClassification * Add FaMTEBClustering * Add FaMTEBPairClassification * Add FaMTEBRetrieval and BEIRFA and FaMTEBSTS * Add FaMTEBSummaryRetrieval * Add FaMTEB to benchmarks * fix benchmark names * temporary fix metadata * Fix dataset revisions * Update SummaryRetrievalEvaluator.py * Update task files * Update task files * add data domain and subtask description * Update AbsTaskSummaryRetrieval and FaMTEBSummaryRetrieval * Update AbsTaskSummaryRetrieval * Add mock task * Update AbsTaskSummaryRetrieval * Update AbsTaskSummaryRetrieval * make lint * Refactor SummaryRetrieval to subclass BitextMining * Add aggregated datasets --------- Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> * Update tasks table * Docs: update docs according to current state (#1870) * update docs * Apply suggestions from code review Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * update readme * Update README.md Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Adding a banner to the new MMTEB leaderboard (#1908) * Adding a banner to the new MMTEB leaderboard * linting * Update mteb/leaderboard/app.py Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * adding reference to mteb arena --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: Filling missing metadata for leaderboard release (#1895) * Update ArxivClusteringS2S.py * fill some metadat for retrieval * fill in the reste of missing metadata * fix metadata * fix climatefever metadata * fix: Added CQADupstack annotations * removed annotation for non-exisitant task * format * Added financial to other financial dataset * Moved ArguAna annotation to derivate datasets --------- Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.6 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: remove SummaryRetrieval as a type (#1915) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: revert rename and add to description (#1918) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * docs: Add sort to domains for task metadata (#1922) Tests currently go into an infinite loop. This should prevent that. * Update tasks table * 1.31.7 Automatically generated by python-semantic-release * docs: Updated citation for mteb(scandinavian) (#1914) fix: Updated citation for mteb(scandinavian) * fix: Add datasets in CodeRAG-Bench (#1595) * add three out of four datasets in CodeRAG-Bench * add verified CodeRAGStackoverflowPostsRetrieval dataset * clean up code and make some comments * fixed lint errors * addressed comments about code-rag datasets: fixed grammar and remove unnessary code and loop * roll back files which is not supposed to change * fixed the comments in split_by_first_newline() and make the methods private by adding a underscore prefix * refactor to use common args * update task descriptions * add entry in benchmarks * correct the alphanumeric order for the dataset * add in tasks.md * add in tasks.md * update task metadata * update importing path * fix lint errors * correct CodeRAG task metadata description field and id for stackoverflow-posts * fix error in test --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * 1.31.8 Automatically generated by python-semantic-release * update __init__ * update generate_imports script for aggregational tasks * add descriptive stats * remove print from script generate_imports * add rest of metadata * fix tests * add todo for test * Revert "fix tests" This reverts commit 7e8be03. * add back check for multilingual * fix imports --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Mehran Sarmadi <128898167+mehran-sarmadi@users.noreply.github.com> Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> Co-authored-by: Wissam Siblini <36303760+wissam-sib@users.noreply.github.com> Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> Co-authored-by: Pengfei He <hepengfe@gmail.com>

update docs

5062e68

Samoed requested review from KennethEnevoldsen, isaac-chung and x-tabdeveloping January 24, 2025 22:09

isaac-chung changed the title ~~feat: update docs according to current state~~ Docs: update docs according to current state Jan 25, 2025

isaac-chung reviewed Jan 25, 2025

View reviewed changes

Samoed and others added 2 commits January 25, 2025 12:17

Apply suggestions from code review

6be88c9

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

Merge branch 'main' into update_docs

d9a5d99

update readme

965a84b

isaac-chung approved these changes Jan 29, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README.md

cf75a8f

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

isaac-chung approved these changes Jan 30, 2025

View reviewed changes

isaac-chung merged commit 7e5d6c8 into main Jan 30, 2025
10 checks passed

isaac-chung deleted the update_docs branch January 30, 2025 10:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs: update docs according to current state#1870

Docs: update docs according to current state#1870
isaac-chung merged 5 commits intomainfrom
update_docs

Samoed commented Jan 24, 2025

Uh oh!

isaac-chung left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

isaac-chung commented Jan 28, 2025

Uh oh!

Samoed commented Jan 28, 2025

Uh oh!

isaac-chung left a comment

Uh oh!

Uh oh!

isaac-chung commented Jan 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Samoed commented Jan 24, 2025

Checklist

Uh oh!

isaac-chung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

isaac-chung commented Jan 28, 2025

Uh oh!

Samoed commented Jan 28, 2025

Uh oh!

isaac-chung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

isaac-chung commented Jan 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants