dataset: add Mars-VL-Pairs bidirectional retrieval by tommasocerruti · Pull Request #5148 · embeddings-benchmark/mteb

tommasocerruti · 2026-08-11T09:18:28Z

Summary

Adds MarsVLPairsT2IRetrieval and MarsVLPairsI2TRetrieval, bidirectional scientific-caption↔Mars-image retrieval. It fills a MOEB gap: MTEB has no planetary-science retrieval task.

Data and metrics

Uses expert-validated refined_caption from SUSTech/Mars-VL-Pairs@1cac8885e481256d3752ad0d3a0f8f9681c5f206. Frozen data: Cerru02/Mars-VL-Pairs-MTEB@f0084ab0ba2f584b15dc72a82502b38ee490f58d.

Recovered 2,249/2,287 images: 2,203 direct, 1 HTTPS, 42 Wayback, and 3 Wikimedia; 38 were unavailable. Removed the lower-resolution item from two resize-equivalent pairs, leaving 2,247 unique one-to-one pairs with provenance. The source declares CC-BY-4.0; external-image rights may vary.

Both tasks rank the full gallery with main score mrr_at_2247.

Results (%)

Model	Direction	R@1	R@10	MRR	MedR
random	T→I	0.05	0.49	0.34	—
random	I→T	0.00	0.45	0.34	—
Qwen3-VL-Embedding-2B	T→I	13.04	29.37	18.68	76
Qwen3-VL-Embedding-2B	I→T	18.51	34.53	23.97	61

The paper reports Qwen T→I 16.97/33.45/22.55/62 and I→T 15.96/33.71/22.24/47 on all 2,287 pairs. Our 2,247-pair run uses MTEB prompts, so it is metric-equivalent, not an exact protocol reproduction.

Dataset checklist

Fills an existing gap: first planetary-science retrieval tasks in MTEB.
Both tasks load and run with the mteb package.
Ran random plus Qwen/Qwen3-VL-Embedding-2B (multimodal substitute for the text-only suggestion).
Qwen is clearly above random and below saturation in both directions.
Kept all 2,247 valid pairs; the dataset is already evaluation-scale.
Compared with the paper above; exact reproduction is impossible after unavailable/ambiguous source rows were removed.

Validation

6,724 practical tests, Ruff, typos, mypy, media decoding, ID/qrel integrity, and descriptive-stat checks pass.

Closes #4955
Related to #4842

Samoed · 2026-08-12T12:13:02Z

+    query_ids = [f"query-{key}" for key in keys]
+    corpus_ids = [f"corpus-{key}" for key in keys]


You can keep query and corpus without prefixes to simplify a bit

Samoed · 2026-08-12T12:13:34Z

+        is_beta=True,
+    )
+    k_values = (1, 3, 5, 10, 20, 100, 1000, _FROZEN_PAIRS)
+    _top_k = _FROZEN_PAIRS


Suggested change

_top_k = _FROZEN_PAIRS

We need this to rank the full gallery, as the official benchmark does, without it, MTEB only retrieves the top 1,000 results.

Samoed · 2026-08-12T12:14:18Z

+        },
+        is_beta=True,
+    )
+    k_values = (1, 3, 5, 10, 20, 100, 1000, _FROZEN_PAIRS)


Suggested change

k_values = (1, 3, 5, 10, 20, 100, 1000, _FROZEN_PAIRS)

k_values = (1, 3, 5, 10, 20, 100, 1000)

I don't think it would be useful, because it will always output 1

True for recall, but MRR@2247 is not always 1, since it depends on the correct match’s rank. I’ll compute full-gallery MRR separately so we don’t generate unnecessary metrics at 2247

Copilot

Pull request overview

Adds Mars-VL-Pairs as a new bidirectional (text↔image) Any2Any retrieval benchmark in MTEB/MOEB, backed by a frozen Hugging Face dataset revision and accompanied by a construction/audit script plus basic task-level tests and descriptive stats.

Changes:

Introduces MarsVLPairsT2IRetrieval and MarsVLPairsI2TRetrieval with a full-gallery MRR main score (mrr_at_2247) derived from a shared frozen pair table.
Adds a dataset-freezing/auditing pipeline (scripts/data/mars_vl_pairs/create_data.py) and documentation for reproducing the frozen dataset build.
Registers tasks, adds unit tests, and commits descriptive statistics JSONs for both directions.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_tasks/test_mars_vl_pairs.py	Adds unit tests to verify shared IDs/qrels and the full-gallery MRR scoring hook.
scripts/data/mars_vl_pairs/README.md	Documents how to construct, audit, and push the frozen Mars-VL-Pairs dataset.
scripts/data/mars_vl_pairs/create_data.py	Implements the auditing, recovery, validation, and freeze-to-HF workflow for image bytes + provenance.
mteb/tasks/retrieval/eng/mars_vl_pairs.py	Adds the two new retrieval tasks and shared loader/scoring logic.
mteb/tasks/retrieval/eng/init.py	Exports/registers the new tasks in the English retrieval task set.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MarsVLPairsT2IRetrieval.json	Adds descriptive statistics for the T→I task split.
mteb/descriptive_stats/Image/Any2AnyRetrieval/MarsVLPairsI2TRetrieval.json	Adds descriptive statistics for the I→T task split.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Samoed · 2026-08-13T05:09:45Z

+        full_gallery_mrr = mrr(qrels, results, (_FROZEN_PAIRS,))[f"MRR@{_FROZEN_PAIRS}"]
+        return {f"mrr_at_{_FROZEN_PAIRS}": fmean(full_gallery_mrr)}


Won't mrr automatically compute across all top_k values?

top_k controls how many results are retrieved, while k_values controls which MRR scores are calculated. Since k_values stops at 1000, we need to calculate MRR@2247 separately.

You can just set k_values to include _FROZEN_PAIRS and this also would solve #5148 (comment)

That would remove the need for the separate MRR calculation, but _top_k is inherited from the base class as 1000 and isn’t recalculated when a task overrides k_values, so only 1000 results would still be retrieved. We would therefore still need _top_k = 2247 and it would also bring back recall_at_2247 = 1 which I removed based on your earlier feedback

If this is metric from paper, then better to use it

Agree, done.

Samoed · 2026-08-13T05:09:55Z

+
+
+class _MarsVLPairsRetrieval(AbsTaskRetrieval):
+    _top_k = _FROZEN_PAIRS


You shoudn't change private values

Got it. The official benchmark computes MRR over all 2247 candidates, and MTEB only retrieves 1000 by default, so matches below rank 1000 would be missing, making the score differ from the official protocol. I used _top_k because the current retrieval task API doesn't expose a public way to change this limit. So I see two options:

Keep _top_k = 2247 to follow the official protocol.

Use MRR@1000, following MTEB’s default but differing from the paper.

Would you prefer the second option, or is there another supported way to request full-gallery retrieval?

Can you remove _top_k = _FROZEN_PAIRS? Also would be easier to just add k_values explicitly to classes, rather than inherit from other class

These two requests conflict with the current API: reproducing the paper requires ranking all 2,247 candidates, but removing _top_k = 2247 makes MTEB retrieve only 1,000. In that case, mrr_at_2247 would not reproduce the paper’s metric. I moved k_values into both classes, but kept _top_k = 2247 for correctness. Is there another supported way to request all 2,247 results?

Submited fix in f761ff9

tommasocerruti · 2026-08-13T12:00:39Z

@Samoed It overwrites the existing _top_k values in subclasses, which is causing the test failures in CI

Samoed · 2026-08-13T16:38:08Z

I run google/siglip2-large-patch16-512 (I think this is from paper ViT-L-16-SigLIP2-512)

and got these results

	MTEB R@10	PAPER R@10
MarsVLPairsI2TRetrieval	0.21762	0.3712
MarsVLPairsT2IRetrieval	0.23676	0.4237

Can you investigate score difference?

# Conflicts: # mteb/abstasks/retrieval.py

dataset: add Mars-VL-Pairs retrieval

7222be8

tommasocerruti marked this pull request as ready for review August 11, 2026 09:37

ayush1298 added the new dataset Issues related to adding a new task or dataset label Aug 11, 2026

Samoed reviewed Aug 12, 2026

View reviewed changes

fix: address Mars-VL-Pairs review feedback

4e7909f

Copilot AI lite review requested due to automatic review settings August 12, 2026 22:58

Copilot started reviewing on behalf of tommasocerruti August 12, 2026 22:59 View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

Comment thread scripts/data/mars_vl_pairs/create_data.py Outdated

fix: require TLS verification for Mars recovery

ab96ce4

Samoed reviewed Aug 13, 2026

View reviewed changes

tommasocerruti and others added 2 commits August 13, 2026 11:33

fix: use standard metrics for full-gallery MRR

c429a1a

fix top k calculation

f761ff9

Samoed added 4 commits August 13, 2026 15:04

fix test

c504014

remove k values from mock

9cca106

fix index

f4a9938

remove files

34f517b

Merge branch 'main' into add-mars-vl-pairs

d98c8e1

# Conflicts: # mteb/abstasks/retrieval.py

		query_ids = [f"query-{key}" for key in keys]
		corpus_ids = [f"corpus-{key}" for key in keys]

	k_values = (1, 3, 5, 10, 20, 100, 1000, _FROZEN_PAIRS)
	k_values = (1, 3, 5, 10, 20, 100, 1000)

		full_gallery_mrr = mrr(qrels, results, (_FROZEN_PAIRS,))[f"MRR@{_FROZEN_PAIRS}"]
		return {f"mrr_at_{_FROZEN_PAIRS}": fmean(full_gallery_mrr)}



		class _MarsVLPairsRetrieval(AbsTaskRetrieval):
		_top_k = _FROZEN_PAIRS

Uh oh!

Conversation

tommasocerruti commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Data and metrics

Results (%)

Dataset checklist

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Samoed Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tommasocerruti commented Aug 13, 2026

Uh oh!

Samoed commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tommasocerruti commented Aug 11, 2026 •

edited

Loading

Samoed Aug 13, 2026 •

edited

Loading