fix: Add dedicated display for RTEB benchmark results

[q275343119]

📌 Live Demo

A working preview of these changes can be viewed here:
https://huggingface.co/spaces/SmileXing/leaderboard

---

Changes Proposed

This Draft PR introduces a dedicated display for the RTEB benchmark results. The changes include:

1. Sidebar Addition: Added a dedicated "RTEB" option to the benchmarks sidebar selector.
2. Data Filtering: Modified the summary and performance per task modules to exclusively use and display data from the RTEB benchmark.
3. Field Removal:
   • Removed the zero-shot field from the summary table.
   • Consequently, removed the zero-shot filter option from the advanced model filters.
4. Ranking Logic: Updated the ranking logic in the summary table to be based on mean(Task) in descending order.
5. Data Source: The results data is now pulled from the embedding-benchmark/results repository, specifically the feat-use-rteb-results branch.

---

Note: This is a work in progress (Draft PR) to showcase the initial implementation for discussion and feedback.

[Samoed]

I don't think that we want always select same models

[q275343119]

You're right, this isn't the ideal approach. Currently, I don't have a method to determine whether a model belongs to RTEB, so I used a hardcoded solution for now. Once the criteria are more defined, I will refactor this into a more elegant solution.

[KennethEnevoldsen]

Hi @q275343119, there are several changes here that could require some discussion. I think it would be easier to discuss them separately (so that a single one doesn't block changes).

I think this is true for 2, 3, 4 (5 is a given)

for 1) it might be a possibility to set it as a flag (that way we can inspect it without adding it to main before it is ready. WDYT?

[q275343119]

Hi @q275343119, there are several changes here that could require some discussion. I think it would be easier to discuss them separately (so that a single one doesn't block changes).

I think this is true for 2, 3, 4 (5 is a given)

for 1) it might be a possibility to set it as a flag (that way we can inspect it without adding it to main before it is ready. WDYT?

Thanks a lot for the suggestion! I’ll add a flag so that the RTEB-related features can be toggled and won’t affect the main display by default.
I’ll also make some adjustments on top of the latest branch, including reverting the ranking logic (as this will be handled later by @fzoll).
In addition, I’ll open a separate issue to discuss the changes (filtering, zero-shot column, ranking, etc.) so that we can agree on the details before creating new draft PRs.

[q275343119]

Hi @q275343119, there are several changes here that could require some discussion. I think it would be easier to discuss them separately (so that a single one doesn't block changes).
I think this is true for 2, 3, 4 (5 is a given)
for 1) it might be a possibility to set it as a flag (that way we can inspect it without adding it to main before it is ready. WDYT?

Thanks a lot for the suggestion! I’ll add a flag so that the RTEB-related features can be toggled and won’t affect the main display by default. I’ll also make some adjustments on top of the latest branch, including reverting the ranking logic (as this will be handled later by @fzoll). In addition, I’ll open a separate issue to discuss the changes (filtering, zero-shot column, ranking, etc.) so that we can agree on the details before creating new draft PRs.

I have opened an issue to discuss the RTEB-specific changes:
#3095

[q275343119]

I add a new startup argument --show_rteb to control the visibility of branch-specific changes.
we can start the project using the following command:

python -m mteb.leaderboard.app --show_rteb

[KennethEnevoldsen]

Hmm, any reason for having this as a separate class?

[q275343119]

As discussed with @fzliu, I've extended the Benchmark class and modified the load_results method to show specific models when RTEB benchmarks are selected. This change implements the requirement for model-specific result filtering.

This can be removed if we decide not to differentiate models in the final implementation.

[KennethEnevoldsen]

Hmm, do you want to curate the models that appear on RTEB, or do you want to filter them differently? (Is there anything wrong with the current filtering?)

[KennethEnevoldsen]

Hmm I understand that you did it this way, but I think I would do it differently to make it consistent across benchmarks. This also makes the approach more expandable in the future:

So my thoughts now are to move it to the benchmark object to have it as:

summary_table = Benchmark.create_summary_table() # default function but can be overwritten

We should be able to do this in a separate PR without RTEB benchmark (to see that everything works as intended), then adding customization for RTEB is simply a matter of replacing the create_summary_table function. I want the per_task_table to be the same across benchmarks.

The long-context retrieval or BRIGHT is a good test case for custom summary tables.

Let me know what you think

[q275343119]

So my thoughts now are to move it to the benchmark object to have it as:

summary_table = Benchmark.create_summary_table() # default function but can be overwritten

That’s a great idea！！！！！

Let me know what you think

I have a couple of follow-up questions:

• Previously, create_tables() returned both summary_table and per_task_table. With the new approach, should I generate summary_table via benchmark.create_summary_table() and per_task_table via create_task_table() separately?

• In #3096
  the discussion around rank and zero-shot still seems undecided. If later we need RTEB benchmarks to handle rank or zero-shot differently, would using inheritance to override create_summary_table() be an acceptable way to implement that customization(something like RTEBBenchmark)?

[KennethEnevoldsen]

I would create them separately

Regarding how to implement changes, I think all of it can be implemented in benchmark.create_summary_table(), which I would probably just do as:

rteb_bench = Benchmark()
rteb_bench.create_summary_table = rteb_create_summary_table

That way, there is no confusion when people suddenly get an RTEBBenchmark instead of a Benchmark.

[q275343119]

Got it. I'll create a separate PR for this great idea

[q275343119]

Hi @KennethEnevoldsen ,here is the PR(#3126)

[q275343119]

Odd, there are still too many commits, please wait while I process them

[Samoed]

I'm not sure about adding RTEB (Retrieval) as separate. Maybe we should convert it to MenuItem too?

[q275343119]

I'm not sure about adding RTEB (Retrieval) as separate. Maybe we should convert it to MenuItem too?

Sure, no problem. The main reasons are that RTEB is quite popular and has some key differences from MTEB, RTEB includes closed datasets. If you have any feedback or suggestions, we can discuss in #3096.
Also, would it be possible to reflect these changes in the UI as well? I mean remove the flag(show_rteb),or set default value to True
@KennethEnevoldsen

[KennethEnevoldsen]

Generally I think this is good to merge as long as default is false

I have a few concerns around naming E.g., "RTEB Retrieval Embedding Benchmark", vs simply using: "Retrieval". I don't think we need to take these now though.

[KennethEnevoldsen]

delete?

[q275343119]

Yes, since RTEB no longer requires special filtering for models, benchmarks that belong to RTEB will now from Benchmark instead of RTEBBenchmark. As a result, the get_rteb_benchmark.py script is no longer necessary.

Originally, this script generated the BENCHMARK_REGISTRY using the following code:

BENCHMARK_REGISTRY = {
    inst.name: inst
    for name, inst in benchmark_module.__dict__.items()
    if isinstance(inst, (Benchmark, RTEBBenchmark))
}

Now, this approach is no longer needed, so I have removed it.

[KennethEnevoldsen]

Some of these are retrieval - it seems a bit weird that they are not in the retrieval section

For now, it is not a problem, as it is just an optional flag for development, but it will be something we need to solve

[KennethEnevoldsen]

I would avoid the duplication on this item - just define and reuse across both (for now)

[q275343119]

I'll modify this part. I will move the benchmarks that are exclusive to RTEB into RTEB_BENCHMARK_ENTRIES.

[q275343119]

Hi @KennethEnevoldsen ,I've made adjustments to the RTEB_BENCHMARK_ENTRIES to remove duplicate Benchmarks. Could you please review it again to see if there are any other issues? If everything looks good, could we merge it into the main branch?

[Samoed]

@q275343119 Can you run linter?

[q275343119]

@q275343119 Can you run linter?

Thanks for the reminder about running lint. I just realized we have a convenient lint command set up in the project. I'll make sure to use it from now on.

[q275343119]

Hi @Samoed ,could you please review it again to see if there any other issues?