dataset: add ChemRxivRetrieval task to ChemTEB benchmark

[HSILA]

This PR introduces the ChemRxivRetrieval dataset, a high-quality, domain-specific retrieval task designed to complement and expand the existing ChemTEB benchmark introduced in #1708.

As one of the authors of ChemTEB, I am proposing this addition to address a specific gap in our current retrieval evaluation. While the existing retrieval tasks in ChemTEB (based on chemical subsets of NQ and HotpotQA) are valuable, they are relatively small and primarily encyclopedic in nature. ChemRxivRetrieval provides a more comprehensive and technically demanding evaluation that reflects real-world article retrieval and question-answering workflows in the chemical sciences.

Dataset Overview

• Paragraph-level retrieval from open-access ChemRxiv papers (up to May 2025).
• Consisting of 75,000 paragraphs from ChemRxiv articles and 5000 synthetic queries generated using Claude 3.7 Sonnet (via Amazon Bedrock) to ensure high linguistic variety and technical depth. Prompts and generation code are available here.
• This dataset serves as a more robust, domain-specific alternative to the existing "subset-style" retrieval tasks, offering a larger scale and higher complexity for evaluating modern embedding models in specialized scientific contexts.

This addition does not replace previous tasks but acts as a critical extension for users requiring evaluation on authentic, literature-based chemical data.

https://arxiv.org/abs/2508.01643

• I have outlined why this dataset is filling an existing gap in mteb
• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command. Results here
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)

[Samoed]

We don't modify existing benchmarks. You can create new version of ChemTEB instead (e.g. ChemTEB(v1.1))

[HSILA]

@Samoed Thanks for your input. I understand, but can we make it so that running this benchmark triggers the latest version during evaluation?

[Samoed]

It's better to create a new version, because adding a new task makes results a bit less reproducible

[HSILA]

@Samoed I got that point. What I mean is, when we run this:
mteb run -m <model_name> -b "ChemTEB"
it runs the latest version, and for the old version we simply use ChemTEB(v1). I was thinking about changing the old one's name to ChemTEB(v1) and the new one's alias to ChemTEB. Does that work?

[Samoed]

Alias would work, but currently your benchmark don't have only ChemTEB as name, so I'm not sure. WDYT @KennethEnevoldsen?

[KennethEnevoldsen]

Hmm, this would break backward compatibility. We can make it such that there is a deprecation warning if you run ChemTEB(v1), but to not break existing code, we can't redirect ChemTEB to the latest version, as this would change the expected behaviour.

I would suggest something like ChemTEB-latest so that people know that it can change when they set it up. I think could be a suggestion (but that would be a seperate issue)

[HSILA]

@KennethEnevoldsen That makes sense. I updated the PR with a new commit. In this new commit, I added a new benchmark with the name ChemTEB(v1.1) and the alias ChemTEB-latest. I also added the alias ChemTEB(v1) to the old benchmark without changing its name (ChemTEB) for backward compatibility. I think it should be okay now. I didn't understand your point about a separate issue; for what should I open a new one?