[MAEB] Merge in main 20250503

.github/workflows/docs.yml

@@ -7,6 +7,9 @@ on:
     branches: [main]
   pull_request:
 
+permissions:
+  contents: write
+
 jobs:
   create-table-on-pr:
     if: github.event_name == 'pull_request'
@@ -32,8 +35,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-        with:
-          token: ${{ secrets.RELEASE }}
 
       - uses: actions/setup-python@v4
         with:
@@ -49,6 +50,8 @@ jobs:
           make build-docs
 
       - name: Push table
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: |
           git config --global user.email "github-actions[bot]@users.noreply.github.com"
           git config --global user.name "github-actions[bot]"
@@ -57,6 +60,7 @@ jobs:
             echo "No changes detected"
           else
             git add docs/tasks.md
-            git commit -m "Update tasks table"
+            git add docs/benchmarks.md
+            git commit -m "Update tasks & benchmarks tables"
             git push
           fi

Makefile

@@ -38,6 +38,7 @@ build-docs:
 	@echo "--- 📚 Building documentation ---"
 	# since we do not have a documentation site, this just build tables for the .md files
 	python docs/create_tasks_table.py
+	python docs/create_benchmarks_table.py
 
 
 model-load-test:

README.md

@@ -78,8 +78,9 @@ The following links to the main sections in the usage documentation.
 | **General** | |
 | [Evaluating a Model](docs/usage/usage.md#evaluating-a-model) | How to evaluate a model |
 | [Evaluating on different Modalities](docs/usage/usage.md#evaluating-on-different-modalities) | How to evaluate image and image-text tasks |
+| [MIEB](docs/mieb/readme.md) | How to run the Massive Image Embedding Benchmark |
 | **Selecting Tasks** | |
-| [Selecting a benchmark](docs/usage/usage.md#selecting-a-benchmark) | How to select and filter tasks |
+| [Selecting a benchmark](docs/usage/usage.md#selecting-a-benchmark) | How to select benchmarks |
 | [Task selection](docs/usage/usage.md#task-selection) | How to select and filter tasks |
 | [Selecting Split and Subsets](docs/usage/usage.md#selecting-evaluation-split-or-subsets) | How to select evaluation splits or subsets |
 | [Using a Custom Task](docs/usage/usage.md#using-a-custom-task) | How to evaluate on a custom task |
@@ -96,7 +97,8 @@ The following links to the main sections in the usage documentation.
 | **Leaderboard** | |
 | [Running the Leaderboard Locally](docs/usage/usage.md#running-the-leaderboard-locally) | How to run the leaderboard locally |
 | [Report Data Contamination](docs/usage/usage.md#annotate-contamination) | How to report data contamination for a model |
-| [Fetching Result from the Leaderboard](docs/usage/usage.md#fetching-results-from-the-leaderboard) | How to fetch the raw results from the leaderboard |
+| [Loading and working with Results](docs/usage/results.md) | How to load and working with the raw results from the leaderboard, including making result dataframes |
+
 
 
 ## Overview
@@ -107,8 +109,8 @@ The following links to the main sections in the usage documentation.
 | 📋 [Tasks]                     | Overview of available tasks                                                         |
 | 📐 [Benchmarks]                | Overview of available benchmarks                                                    |
 | **Contributing**               |                                                                                     |
-| 🤖 [Adding a model]            | Information related to how to submit a model to MTEB and to the leaderboard         |
-| 👩‍🔬 [Reproducible workflows]    | Information related to how to create reproducible workflows with MTEB               |
+| 🤖 [Adding a model]            | How to submit a model to MTEB and to the leaderboard                                |
+| 👩‍🔬 [Reproducible workflows]    | How to create reproducible workflows with MTEB                                      |
 | 👩‍💻 [Adding a dataset]          | How to add a new task/dataset to MTEB                                               |
 | 👩‍💻 [Adding a benchmark]        | How to add a new benchmark to MTEB and to the leaderboard                           |
 | 🤝 [Contributing]              | How to contribute to MTEB and set it up for development                             |
@@ -172,3 +174,4 @@ Some of these amazing publications include (ordered chronologically):
 - Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li. "[LongEmbed: Extending Embedding Models for Long Context Retrieval](https://arxiv.org/abs/2404.12096)" arXiv 2024
 - Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, Kristoffer Laigaard Nielbo. "[The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://arxiv.org/abs/2406.02396)" arXiv 2024
 - Ali Shiraee Kasmaee, Mohammad Khodadad, Mohammad Arshi Saloot, Nick Sherck, Stephen Dokas, Hamidreza Mahyar, Soheila Samiee. "[ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain](https://arxiv.org/abs/2412.00532)" arXiv 2024
+- Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, Niklas Muennighoff. "[MIEB: Massive Image Embedding Benchmark](https://arxiv.org/abs/2504.10471)" arXiv 2025