refactor: split BRIGHT benchmark into individual subset tasks

[whybe-choi]

Close #3268

This pull request adds new BRIGHT subset benchmarks and their corresponding descriptive statistics to the retrieval benchmark suite. These changes enable more granular, domain-specific evaluation for reasoning-intensive retrieval tasks, both for standard and long document formats.

Benchmark additions

• Introduced two new benchmarks, BRIGHT_SUBSETS and BRIGHT_SUBSETS_LONG, to the mteb/benchmarks/benchmarks/benchmarks.py file, covering individual domains of the BRIGHT benchmark for both standard and long document retrieval tasks. [1] [2]
• Registered the new benchmarks in the mteb/benchmarks/benchmarks/__init__.py file for import and usage. [1] [2]

Descriptive statistics

• Added descriptive statistics JSON files for each new BRIGHT subset retrieval task, including both standard and long formats (e.g., BrightBiologyRetrieval.json, BrightBiologyLongRetrieval.json, etc.), detailing sample counts, text lengths, and relevant document statistics for each domain. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

Minor improvement

• Minor formatting fix in the BEIR_NL benchmark description for improved readability.

[KennethEnevoldsen]

Hmm, this change will invalidate all previous results on BRIGHT.

You know that you can also simply subselect from a task using:

task = mteb.get_task("BrightRetrieval", eval_splits=..., hf_subet=...)

For the leaderboard display it is even possible to create custom summary tables (see e.g. #3272)

[Samoed]

You know that you can also simply subselect from a task using:

Yes, but BRIGHT requires different prompts for different subsets, and because of that we probably need to split it. We can add support to configure prompts for different subsets, but I'm not sure if it good idea

[KennethEnevoldsen]

Yes, but BRIGHT requires different prompts for different subsets, and because of that we probably need to split it. We can add support to configure prompts for different subsets, but I'm not sure if it good idea

Ohh... Yeah that is hard to fix.

I see that the original BRIGHT(long) only has four models and BRIGHT only has 12, so I guess it is possible to rerun them

[Muennighoff]

If the scores change, are the new scores more similar or more different from the official scores? If closer then I think it is fine & maybe we can rerun some models. I think that for many models on our BRIGHT leaderboard I just converted the scores from https://brightbenchmark.github.io/ to MTEB format when we originally added so they may be still fine if these changes actually make our implementation closer to that one.

[whybe-choi]

Would it be enough to evaluate the performance of ReasonIR, or is there a list of other models that would be good enough to test?

[Samoed]

To check implementation, this will be enough, just don't update old leaderboard

[whybe-choi]

After split BrightRetrieval into multiple tasks, I ran ReasonIR on them with task-specific prompts using the following code:

import torch
import mteb

# https://github.com/facebookresearch/ReasonIR/tree/main/evaluation/bright/configs/reasonir
prompts_dict = {
    "BrightBiologyRetrieval": "Given a Biology post, retrieve relevant passages that help answer the post",
    "BrightEarthScienceRetrieval": "Given a Earth Science post, retrieve relevant passages that help answer the post",
    "BrightEconomicsRetrieval": "Given a Economics post, retrieve relevant passages that help answer the post",
    "BrightPsychologyRetrieval": "Given a Psychology post, retrieve relevant passages that help answer the post",
    "BrightRoboticsRetrieval": "Given a Robotics post, retrieve relevant passages that help answer the post",
    "BrightStackoverflowRetrieval": "Given a Stackoverflow post, retrieve relevant passages that help answer the post",
    "BrightSustainableLivingRetrieval": "Given a Sustainable Living post, retrieve relevant passages that help answer the post",
    "BrightPonyRetrieval": "Given a Pony question, retrieve relevant passages that help answer the question",
    "BrightLeetcodeRetrieval": "Given a coding problem, retrieve relevant examples that help answer the problem",
    "BrightAopsRetrieval": "Given a Math problem, retrieve relevant examples that help answer the problem",
    "BrightTheoremQATheoremsRetrieval": "Given a Math problem, retrieve relevant theorems that help answer the problem",
    "BrightTheoremQAQuestionsRetrieval": "Given a Math problem, retrieve relevant examples that help answer the problem",
}

tasks = mteb.get_tasks(tasks=list(prompts_dict.keys()), languages=["eng"])
evaluation = mteb.MTEB(tasks=tasks)

model = mteb.get_model(
    "ReasonIR/ReasonIR-8B",
    model_kwargs={"torch_dtype": torch.bfloat16},
    prompts_dict=prompts_dict,
)

evaluation.run(
    model,
    save_predictions=True,
    output_folder="evaluation/results",
    encode_kwargs={"batch_size": 1},
)

The results are as follows:

	Bio.	Earth.	Econ.	Psy.	Rob.	Stack.	Sus.	Leet.	Pony	AoPS	TheoQ.	TheoT.	Avg.
before split	24.31	30.83	24.27	28.95	18.40	21.68	20.57	18.14	9.49	4.84	18.21	26.42	20.51
after split	26.18	30.71	23.96	29.76	18.62	21.15	19.89	19.65	9.22	5.12	18.34	27.12	20.81

In the paper:

[Samoed]

Great results! But I'm a bit unsure does prompts applied correctly when they're passing thought get_model?

[whybe-choi]

mteb/mteb/models/instruct_wrapper.py

Lines 158 to 171 in d2c704c

	if instruction:
	logger.info(f"Using instruction: '{instruction}' for task: '{task_name}'")
	embeddings = self.model.encode(
	sentences,
	prompt=instruction,
	**kwargs,
	)
	if isinstance(embeddings, torch.Tensor):
	# sometimes in kwargs can be return_tensors=True
	embeddings = embeddings.cpu().detach().float().numpy()
	return embeddings

After adding code to print the instruction inside the code, the following output was produced:

# Biology
Retrieval
    - BrightBiologyRetrieval, s2p


instruction: <|user|>
Given a Biology post, retrieve relevant passages that help answer the post<|embed|>

Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 103/103 [00:06<00:00, 15.80it/s]
instruction: <|embed|>

Batches:   0%|                                                                                            | 2/50000 [00:02<18:01:38,  1.30s/it

# Psychology
Retrieval
    - BrightPsychologyRetrieval, s2p


instruction: <|user|>
Given a Psychology post, retrieve relevant passages that help answer the post<|embed|>

Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 101/101 [00:07<00:00, 14.12it/s]
instruction: <|embed|>

Batches:   0%|                                                                                                       | 0/50000 [00:01<?, ?it/s]

# Aops
Retrieval
    - BrightAopsRetrieval, s2p


instruction: <|user|>
Given a Math problem, retrieve relevant examples that help answer the problem<|embed|>

Batches: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 111/111 [00:06<00:00, 16.13it/s]
instruction: <|embed|>

Batches:   0%|                                                                                            | 17/50000 [00:09<7:16:33,  1.91it/s]

[Samoed]

Interesting, thanks! I didn’t think that would work since it’s a bit unintended, but maybe we should update the code to handle this case.

I've checked code for ReasonIR and found some other places that can help to reproduce:

1. For some cases, rewritten query is concatenated with query https://github.com/facebookresearch/ReasonIR/blob/0aac96269e455965949df16520fab72da68ffc22/evaluation/bright/run.py#L82-L87
2. Sometimes reason trases added to the query https://github.com/facebookresearch/ReasonIR/blob/0aac96269e455965949df16520fab72da68ffc22/evaluation/bright/run.py#L124
3. Maybe ids can be filtered (ref Excluded IDs missing from BRIGHT dataset #2696) but in ReasonIR code they're just check that no ids are intersect https://github.com/facebookresearch/ReasonIR/blob/0aac96269e455965949df16520fab72da68ffc22/evaluation/bright/run.py#L130-L131

@Muennighoff Can you help what we can do to reproduce results?

[Muennighoff]

I think the IDs filtering is probably the main missing piece to fully reproduce results?

[whybe-choi]

I think points 1 and 2 are a separate issue, as they are related to query expansion. The problem of the performance not being reproducible in the single ReasonIR model seems to be related to the issue mentioned in point 3.

[whybe-choi]

@Samoed

I think it would be better to close this PR and work on it later together with Excluded IDs missing from BRIGHT dataset #2696. Also, should revise it to fit the v2 format and include descriptive stats as well. What do you think?

[Samoed]

I think it would be better to close this PR and work on it later together

Do you mean that you don't want tasks in this pr and will add another PR for #2696?

Also, should revise it to fit the v2 format and include descriptive stats as well. What do you think?

Yes, you need to add statistic to merge. To apply v2 format, you can select subsets from https://huggingface.co/datasets/mteb/BrightRetrieval, but retrieval dataset loader reqired dataset to have strictly corpus, qrels and quries, maybe we need to reupload them instead

[whybe-choi]

What tasks need to be redone for this PR? I'm confused about the changes with the v2 format, so I would appreciate your help.

[Samoed]

I think we can solve #2696 in this pr, because otherwise we would need to create v2 versions of these tasks, which I think is not good solution

[Samoed]

To follow v2 format, you can remove conversion dataset to dict and pass dataset directly.

domain_corpus_long = domain_corpus_long.rename_column("content", "text")
queries = queries.rename_column("query", "text")
...
return domain_corpus_long, queires, relevant_docs

[Samoed]

And then here it should look like

self.dataset["default"]["long"]["corpus"], self.dataset["default"]["long"]["queries"], self.dataset["default"]["long"]["relevant_documents"]

You can refer to

mteb/mteb/abstasks/retrieval_dataset_loaders.py

Lines 25 to 38 in 0ead029

	class RetrievalSplitData(TypedDict):
	"""A dictionary containing the corpus, queries, relevant documents, instructions, and top-ranked documents for a retrieval task.
	Attributes:
	corpus: The corpus dataset containing documents. Should have columns `id`, `title`, `text` or `image`.
	queries: The queries dataset containing queries. Should have columns `id`, `text`, `instruction` (for instruction retrieval/reranking) or `image`.
	relevant_docs: A mapping of query IDs to relevant document IDs and their relevance scores. Should have columns `query-id`, `corpus-id`, `score`.
	top_ranked: A mapping of query IDs to a list of top-ranked document IDs. Should have columns `query-id`, `corpus-ids` (list[str]). This is optional and used for reranking tasks.
	"""
	corpus: CorpusDatasetType
	queries: QueryDatasetType
	relevant_docs: RelevantDocumentsType
	top_ranked: TopRankedDocumentsType | None

[Samoed]

I run with bm25 and got these results. Overall they're showing same difference as in #3285 (comment). So problem totally in data somewhere. I will continue debugging later

task_name	bm25s (mteb)	bm25 (bright)	diff	matching
BrightAopsRetrieval	0.03155	0.0498	0.0182	-
BrightBiologyRetrieval	0.08034	0.0774	0.0029	-
BrightEarthScienceRetrieval	0.12803	0.1283	0.0003	+
BrightEconomicsRetrieval	0.1033	0.1026	0.0007	+
BrightLeetcodeRetrieval	0.14568	0.2532	0.1075	-
BrightPonyRetrieval	0.02222	0.0226	0.0004	+
BrightPsychologyRetrieval	0.07645	0.0765	0.0001	+
BrightRoboticsRetrieval	0.06229	0.0623	0.0000	+
BrightStackoverflowRetrieval	0.14359	0.1636	0.0200	-
BrightSustainableLivingRetrieval	0.08385	0.0851	0.0013	-
BrightTheoremQAQuestionsRetrieval	0.05413	0.0735	0.0194	-
BrightTheoremQATheoremsRetrieval	0.01566	0.0051	0.0106	-
BrightBiologyLongRetrieval	0.09547	0.0955	0.0000	+
BrightEarthScienceLongRetrieval	0.17816	0.1782	0.0000	+
BrightEconomicsLongRetrieval	0.15372	0.1537	0.0000	+
BrightPonyLongRetrieval	0.02496	0.0250	0.0000	+
BrightPsychologyLongRetrieval	0.1703	0.1703	0.0000	+
BrightRoboticsLongRetrieval	0.05941	0.0594	0.0000	+
BrightStackoverflowLongRetrieval	0.23504	0.2350	0.0000	+
BrightSustainableLivingLongRetrieval	0.18519	0.1852	0.0000	+

Script to use bm25s in BRIGHT

def retrieval_bm25(queries, query_ids, documents, doc_ids, excluded_ids, long_context, **kwargs):
    import bm25s
    import Stemmer

    stemmer_language = 'english'
    stopwords = 'en'
    stemmer = Stemmer.Stemmer(stemmer_language)
    encoded_corpus = bm25s.tokenize(documents, stopwords=stopwords, stemmer=stemmer)

    retriever = bm25s.BM25()
    retriever.index(encoded_corpus)

    corpus_idx_to_id = {i: doc_id for i, doc_id in enumerate(doc_ids)}
    query_token_strs = bm25s.tokenize(queries, stopwords=stopwords, stemmer=stemmer)

    queries_results, queries_scores = retriever.retrieve(
        query_token_strs,
        k=min(1000, len(corpus_idx_to_id))
    )

    all_scores = {}
    for qi, query_id in enumerate(query_ids):
        query_results = queries_results[qi]
        scores = queries_scores[qi]
        all_scores[str(query_id)] = {}
        for ri in range(len(query_results)):
            doc_idx = query_results[ri]
            score = scores[ri]
            doc_id = corpus_idx_to_id[doc_idx]
            all_scores[str(query_id)][str(doc_id)] = float(score)
        for did in set(excluded_ids[str(query_id)]):
            if did != "N/A" and did in all_scores[str(query_id)]:
                all_scores[str(query_id)].pop(did)
        cur_scores = sorted(all_scores[str(query_id)].items(), key=lambda x: x[1], reverse=True)[:1000]
        all_scores[str(query_id)] = {pair[0]: pair[1] for pair in cur_scores}
    return all_scores

[Samoed]

I found a source of problem

mteb/mteb/abstasks/retrieval.py

Lines 344 to 348 in 42dea01

	data_split["relevant_docs"], data_split["queries"] = (
	_filter_queries_without_positives(
	data_split["relevant_docs"], data_split["queries"]
	)
	)

If I remove filtering, then I'm getting the same metrics for bm25. Should we can option to disable filtering to reproduce old results, or we just leave as is? WDYT @KennethEnevoldsen @Muennighoff?

[KennethEnevoldsen]

If I remove filtering, then I'm getting the same metrics for bm25. Should we can option to disable filtering to reproduce old results, or we just leave as is? WDYT @KennethEnevoldsen @Muennighoff?

Wouldn't we rather reproduce the old results and then potentially version bump it and add the filtering?

[Samoed]

I've run tasks and without this filtering I can reproduce scores from paper. I'm not sure how version bump would help us

[KennethEnevoldsen]

So, if I understand correctly, we have the choices between reproducing paper scores or the previous implementation?

Then I would target the paper and relabel it as a bugfix.

[Samoed]

I rerun BRIGHT and now results are matching!

task_name	bm25s (mteb)	bm25 (bright)
BrightAopsRetrieval	0.0498	0.0498
BrightBiologyRetrieval	0.07736	0.0774
BrightEarthScienceRetrieval	0.12825	0.1283
BrightEconomicsRetrieval	0.10258	0.1026
BrightLeetcodeRetrieval	0.25321	0.2532
BrightPonyRetrieval	0.02264	0.0226
BrightPsychologyRetrieval	0.07645	0.0765
BrightRoboticsRetrieval	0.06229	0.0623
BrightStackoverflowRetrieval	0.16358	0.1636
BrightSustainableLivingRetrieval	0.08507	0.0851
BrightTheoremQAQuestionsRetrieval	0.07349	0.0735
BrightTheoremQATheoremsRetrieval	0.00509	0.0051
BrightBiologyLongRetrieval	0.09547	0.0955
BrightEarthScienceLongRetrieval	0.17816	0.1782
BrightEconomicsLongRetrieval	0.15372	0.1537
BrightPonyLongRetrieval	0.02496	0.0250
BrightPsychologyLongRetrieval	0.1703	0.1703
BrightRoboticsLongRetrieval	0.05941	0.0594
BrightStackoverflowLongRetrieval	0.23504	0.2350
BrightSustainableLivingLongRetrieval	0.18519	0.1852

I changed:

• Updated revision of datasets to the latest. Seems that our revision was beteween data uploading and datataset wasn't fully finish transition. theoremqa_theorems was updated after this commit
• bm25 didn't support reranking tasks (used all documents for searching)

Also I've splitted BRIGHT v1.1 on short and long subsets, but maybe we can convert them back. I don't know why I thought that problem was in filtering. I will try to evalute bge model to check instructions

[KennethEnevoldsen]

Shouldn't we just combine this into one table with both long and short as two different columns (we can also have different columns for the different domains)

[KennethEnevoldsen]

Feel free to delete the benchmark here and add that in a separate PR.

[Samoed]

Short Bright

MTEB

task_name	BrightBiologyRetrieval	BrightEarthScienceRetrieval	BrightEconomicsRetrieval	BrightPsychologyRetrieval	BrightRoboticsRetrieval	BrightStackoverflowRetrieval	BrightSustainableLivingRetrieval	BrightLeetcodeRetrieval	BrightPonyRetrieval	BrightAopsRetrieval	BrightTheoremQAQuestionsRetrieval	BrightTheoremQATheoremsRetrieval
BAAI/bge-large-en-v1.5	0.1167	0.24558	0.16605	0.17464	0.11713	0.1083	0.13326	0.26681	0.05724	0.06	0.13057	0.069
sentence-transformers/all-mpnet-base-v2	0.15098	0.20414	0.16639	0.22664	0.08221	0.11024	0.15343	0.26404	0.06996	0.05325	0.20035	0.10779

Bright

Model	BrightBiologyRetrieval	BrightEarthScienceRetrieval	BrightEconomicsRetrieval	BrightPsychologyRetrieval	BrightRoboticsRetrieval	BrightStackoverflowRetrieval	BrightSustainableLivingRetrieval	BrightLeetcodeRetrieval	BrightPonyRetrieval	BrightAopsRetrieval	BrightTheoremQAQuestionsRetrieval	BrightTheoremQATheoremsRetrieval
BAAI/bge-large-en-v1.5	11.7	24.6	16.6	17.5	11.7	10.8	13.3	26.7	5.7	6.0	13.0	6.9
sentence-transformers/all-mpnet-base-v2	15.1	20.4	16.6	22.7	8.2	11.0	15.3	26.4	7.0	5.3	20.0	10.8

Long Bright

MTEB

task_name	BrightBiologyLongRetrieval	BrightEarthScienceLongRetrieval	BrightEconomicsLongRetrieval	BrightPsychologyLongRetrieval	BrightRoboticsLongRetrieval	BrightStackoverflowLongRetrieval	BrightSustainableLivingLongRetrieval	BrightPonyLongRetrieval
BAAI/bge-large-en-v1.5	0.16424	0.2773	0.20874	0.11584	0.10891	0.13248	0.16898	0.0036
sentence-transformers/all-mpnet-base-v2	0.25566	0.34052	0.18932	0.15842	0.10891	0.14957	0.18009	0.0119

Bright

Model	BrightBiologyLongRetrieval	BrightEarthScienceLongRetrieval	BrightEconomicsLongRetrieval	BrightPsychologyLongRetrieval	BrightRoboticsLongRetrieval	BrightStackoverflowLongRetrieval	BrightSustainableLivingLongRetrieval	BrightPonyLongRetrieval
BAAI/bge-large-en-v1.5	16.4	27.7	20.9	11.6	10.9	13.3	16.9	0.4
sentence-transformers/all-mpnet-base-v2	25.6	34.1	18.9	15.8	10.9	15.0	18.0	1.2

[KennethEnevoldsen]

I believe we are good to merge here! Great job everyone.