dataset: add MixBench mixed-modality retrieval by tommasocerruti · Pull Request #5144 · embeddings-benchmark/mteb

tommasocerruti · 2026-08-11T09:02:00Z

Summary

Adds four Any2AnyRetrieval tasks—MixBenchMSCOCO, MixBenchGoogleWIT, MixBenchVisualNews, and MixBenchOVEN—and the MIXBENCH group. It fills a MOEB gap: retrieval where text-only, image-only, and image+text documents compete in one pool. Combined rows reach encoders as one multimodal input.

Data and metrics

Uses mixed-modality-search/MixBench2026@17a9e705b2346b118a63f163f10e47325f9e9ecc. Exact qrels come from MixBench25@88e3916036ea0bdb205f4da885d6e947a565c1a0; the current release omits them. No derivative dataset is used.

Counts (queries/corpus/qrels): MSCOCO 984/984/984; WIT 1000/4421/1000; VisualNews 981/981/981; OVEN 1000/1000/1000. Corpora preserve the near-even text/image/image+text mix. Main score: ndcg_at_10.

Results

Model	MSCOCO	WIT	VisualNews	OVEN
`mteb/baseline-random-encoder`	.00599	.00030	.00663	.00528
`Qwen/Qwen3-VL-Embedding-2B`	.88198	.79488	.85505	.54664
Paper: VLM2Vec-Qwen2VL-7B	.753	.632	.734	.412

Qwen is above random and below saturation. The paper row is not an exact reproduction: it uses another checkpoint, and public WIT has 4421 documents versus 4423 reported.

Dataset checklist

I outlined why this dataset fills an existing gap in mteb.
I tested all four tasks with the mteb package.
I ran models and added results above:
- mteb/baseline-random-encoder
- Qwen/Qwen3-VL-Embedding-2B (multimodal replacement for the suggested text-only model)
Performance is neither trivial nor random.
I retained the evaluation-scale subsets; no downsampling was needed.
I added paper scores; exact reproduction is not applicable for the reasons above.

Validation

make lint, make typecheck, focused tests, and the full suite pass. Every qrel resolves and image decodes. Mixed-media licensing is heterogeneous; metadata uses not specified.

Closes #4953
Related to #4842
Blocked by #4159

Samoed · 2026-08-12T12:19:23Z

+            corpus_col_inputs["text"] = [
+                text
+                for text in corpus.map(_corpus_to_dict)["text"]
+                if text is not None and text.strip()
+            ]


Why these changes needed?

MixBench rows can contain text, an image, or both. This filters out empty values so models only receive the modalities present in each row.

Samoed · 2026-08-12T12:20:07Z

+  year = {2025},
+}
+""",
+    contacts=["tommasocerruti"],


There shoud be authors of benchmarks

Suggested change

contacts=["tommasocerruti"],

contacts=None,

Samoed · 2026-08-12T12:20:32Z

-        text_embeddings = []
-        image_embeddings = []
-        audio_embeddings = []
-        video_embeddings = []
-
-        if "text" in batch:
-            text_embeddings = [
-                _string_to_vector(txt, embedding_dim) for txt in batch["text"]
-            ]
-        if "image" in batch:
-            image_embeddings = [
-                _image_to_vector(img, embedding_dim) for img in batch["image"]
-            ]
-        if "audio" in batch:
-            audio_embeddings = [
-                _audio_to_vector(audio, embedding_dim) for audio in batch["audio"]
-            ]
-        if "video" in batch:
-            video_embeddings = [
-                _video_to_vector(
-                    video,
-                    embedding_dim,
-                )
-                for video in batch["video"]
-            ]
-
-        # Combine embeddings
-        max_len = max(
-            [
-                len(text_embeddings),
-                len(image_embeddings),
-                len(audio_embeddings),
-                len(video_embeddings),
-            ]
-        )
-        for i in range(max_len):
-            combined_embedding = np.zeros(embedding_dim, dtype=np.float32)
-            count = 0
-            for embeddings_list in [
-                text_embeddings,
-                image_embeddings,
-                audio_embeddings,
-                video_embeddings,
-            ]:
-                if i < len(embeddings_list):
-                    combined_embedding += embeddings_list[i]
-                    count += 1
-            if count > 0:
-                combined_embedding /= count
-            embeddings.append(combined_embedding)


Why this is needed?

MixBench needs the baseline to skip missing modalities, but I agree this rewrite is too broad. I’ll preserve the existing logic, add only the missing-value checks, and test that normal inputs produce exactly the same embeddings as before. Would that be fine? @Samoed

Samoed · 2026-08-12T12:21:53Z

+    num_queries_with_text: NotRequired[int]
+    num_queries_with_image: NotRequired[int]
+    num_queries_with_audio: NotRequired[int]
+    num_queries_with_video: NotRequired[int]


This should be just equal num_queries or num_documents

These counts are used for per-modality statistics, and can differ from the total (e.g., MSCOCO has 984 documents, but only 656 contain text and 656 contain an image).

Samoed · 2026-08-12T12:23:38Z

Seems some corpus and queries just have empty image column. In that case would be easier to just drop during load, rather than changing models and other parts of code

I can drop columns that are empty for the whole split. However, corpus rows are mixed (some have text, some images, and some both), so we still need to filter empty values per row.

We don't support this kind of input for now

Copilot

Pull request overview

Adds the MixBench mixed-modality retrieval benchmark to MTEB/MOEB, enabling evaluation where text-only, image-only, and image+text documents compete in a single retrieval pool. The PR also updates model/dataloader plumbing and descriptive-statistics logic to correctly handle sparsely populated modalities (e.g., some rows missing text or images).

Changes:

Introduces four new Any2AnyRetrieval tasks (MSCOCO, Google WIT, VisualNews, OVEN) plus the MIXBENCH benchmark group.
Updates multimodal batching, random baseline embedding combination, dataloader corpus handling, and retrieval descriptive statistics to support missing-per-row modalities.
Adds targeted tests and descriptive stats fixtures for MixBench and mixed-modality behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_tasks/test_task_quality.py	Exempts specific MixBench subsets from generic quality checks; adjusts unique-count denominator logic for sparse modalities.
tests/test_tasks/test_mixbench.py	Adds tests covering mixed-modality batching, dataloader behavior, instruct wrapper, random baseline, loader normalization, and sparse-modality stats.
scripts/data/mixbench/validate.py	Adds a validation script to audit pinned MixBench payloads and original qrels consistency.
mteb/types/statistics.py	Extends `RetrievalDescriptiveStatistics` to optionally include per-modality populated-count fields.
mteb/tasks/retrieval/eng/mixbench_retrieval.py	Implements MixBench task loaders, normalization, qrels recovery from MixBench25, and task metadata.
mteb/tasks/retrieval/eng/init.py	Exposes the new MixBench retrieval tasks via the English retrieval tasks module.
mteb/models/sentence_transformer_wrapper.py	Updates multimodal batch construction to omit missing per-sample modality values and reject fully-empty samples.
mteb/models/model_implementations/random_baseline.py	Updates random baseline to ignore missing modalities when averaging per-sample embeddings.
mteb/models/instruct_wrapper.py	Reuses shared multimodal batching logic to preserve only present modalities per sample.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MixBenchMSCOCO.json	Adds descriptive stats snapshot for MixBenchMSCOCO.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MixBenchGoogleWIT.json	Adds descriptive stats snapshot for MixBenchGoogleWIT.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MixBenchVisualNews.json	Adds descriptive stats snapshot for MixBenchVisualNews.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MixBenchOVEN.json	Adds descriptive stats snapshot for MixBenchOVEN.
mteb/benchmarks/benchmarks/benchmarks.py	Adds the `MIXBENCH` benchmark definition aggregating the four tasks.
mteb/benchmarks/benchmarks/init.py	Exports `MIXBENCH` from the benchmarks package.
mteb/abstasks/retrieval.py	Updates retrieval descriptive-statistics calculation to ignore `None` rows and record per-modality populated counts when sparse.
mteb/_create_dataloaders.py	Updates corpus text construction to tolerate missing text and allows `text` to be `None` in collate for mixed-modality rows.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Samoed · 2026-08-13T10:59:01Z

Can you revert changes other than tasks code? To fully support partly evaluation of modalities we need more work and this is not scope of this PR

tommasocerruti · 2026-08-13T19:08:39Z

Ok I’ll keep this PR limited to the MixBench task code. Since MixBench requires per-row missing modalities, should I open a separate issue/PR for that support and keep this PR blocked until it lands, or what do you suggest? @Samoed

Samoed · 2026-08-13T19:24:06Z

You can update this PR and just create an issue for this dataset

tommasocerruti · 2026-08-13T19:29:23Z

Just to confirm: after reverting the framework changes, MixBench can be added as a beta task but cannot be evaluated faithfully until the sparse-modality issue is resolved. Should this PR remain blocked until then, or should the beta task definitions be merged first?

Samoed · 2026-08-13T19:42:49Z

but cannot be evaluated faithfully until the sparse-modality issue is resolved

Why so?

tommasocerruti · 2026-08-13T19:52:55Z

Because it has missing modalities per row, not per split. It's the same limitation already described in #4159

Samoed · 2026-08-13T19:57:58Z

I thought only one task task from these reqired this format. In that case we can't integrate these tasks

tommasocerruti · 2026-08-13T20:04:45Z

@AdnanElAssadi56, do you agree that MixBench cannot currently be integrated? It was approved in the main MOEB issue, but all four subsets require per-row optional modalities. This limitation is already tracked in #4159, and this PR includes a focused implementation with regression tests.

Samoed · 2026-08-13T20:33:34Z

We can't merge implementaation like this

tommasocerruti · 2026-08-13T20:50:35Z

@Samoed, please clarify why the current implementation cannot be merged together with the fix for the framework limitation. MixBench was approved for MOEB, and I’ve invested significant effort implementing and testing it. Feedback such as

We can't merge implementation like this

is too vague to constructively act on, you should clarify better the design you expect.
@AdnanElAssadi56, I’d appreciate your perspective as well.

Samoed · 2026-08-14T01:16:24Z

Currently we're expecting that all rows in columns always contain data. We do not support datasets with sparse inputs, adding it's support would require a lot of work which is out of scope of this pr

tommasocerruti added 4 commits August 11, 2026 02:42

dataset: add MixBench mixed-modality retrieval

ede74c9

dataset: load only MixBench retrieval parquets

73541a8

dataset: scope sparse retrieval statistics

cf9c34c

fix: preserve instruct wrapper modality forwarding

ab52d52

tommasocerruti marked this pull request as ready for review August 11, 2026 09:28

ayush1298 added the new dataset Issues related to adding a new task or dataset label Aug 11, 2026

Samoed requested changes Aug 12, 2026

View reviewed changes

fix: address MixBench sparse modality review

8348b79

Copilot AI lite review requested due to automatic review settings August 12, 2026 23:17

Copilot started reviewing on behalf of tommasocerruti August 12, 2026 23:17 View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

Comment thread scripts/data/mixbench/validate.py

Comment thread mteb/models/sentence_transformer_wrapper.py

fix: address MixBench Copilot review

99b87a2

tommasocerruti requested a review from Samoed August 13, 2026 10:51

tommasocerruti mentioned this pull request Aug 13, 2026

Allow for interleave modality dataset #4159

Open

Uh oh!

Conversation

tommasocerruti commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Data and metrics

Results

Dataset checklist

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tommasocerruti commented Aug 11, 2026 •

edited

Loading

Samoed commented Aug 14, 2026 •

edited

Loading