Skip to content

fix: Update RTEB summary columns #3226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KennethEnevoldsen merged 20 commits into from
Sep 30, 2025
Merged

fix: Update RTEB summary columns #3226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
82d9e29
fix(models): ensure prompt_type is passed to format_instruction (#3216)
whybe-choi Sep 26, 2025
d0d427d
1.38.58
Sep 27, 2025
08bba49
Adding Cohere's output_dimension and embedding_type parameter (#3204)
fzoll Sep 27, 2025
e863bc1
dataset: add swedish cpc patent classifications to mteb (#3072)
Atheer2104 Sep 27, 2025
8c180d4
fix: AttributeError in ColPaliEngineWrapper similarity method (#3177)
FacerAin Sep 27, 2025
0aacba4
Update tasks & benchmarks tables
github-actions[bot] Sep 27, 2025
2e292cf
1.38.59
Sep 27, 2025
f58ac2b
fix: prevent EOS token truncation (#3218)
whybe-choi Sep 27, 2025
3e86531
1.38.60
Sep 27, 2025
15f9909
Update giga embeddings (#3210)
ekolodin Sep 29, 2025
cb03bd4
fix: Refactor split create_tables into static Benchmark methods (#3126)
q275343119 Sep 29, 2025
a52723a
1.38.61
Sep 29, 2025
4f58684
Extending the RTEB benchmark (#3223)
fzoll Sep 29, 2025
7f5990a
Update tasks & benchmarks tables
github-actions[bot] Sep 29, 2025
e196ab9
feat - filter_by_privacy
q275343119 Sep 30, 2025
3fa3ccd
feat - add new fields for rteb part
q275343119 Sep 30, 2025
cefd826
feat - getattr
q275343119 Sep 30, 2025
fb5902a
feat - adjust privacy filter logic
q275343119 Sep 30, 2025
4dec6d4
feat - enhance summary table column renaming and add 'is_public' fiel…
smile-pinai Sep 30, 2025
73f144c
fix: remove unused 'is_public' attribute from TaskResult
SmileLikeYe Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ The following table gives you an overview of the benchmarks in MTEB.
| RTEB(Code, beta) | RTEB Code | 8 | Retrieval: 8 | [Programming, Written] | eng,go,javascript,jpn,python,sql |
| RTEB(Health, beta) | RTEB Healthcare | 4 | Retrieval: 4 | [Academic, Medical, Written] | deu,eng,fra,spa |
| RTEB(Law, beta) | RTEB Legal | 7 | Retrieval: 7 | [Legal, Written] | deu,eng,fra,jpn |
| RTEB(beta) | RTEB Retrieval Embedding Benchmark | 28 | Retrieval: 28 | [Academic, Encyclopaedic, Financial, Legal, Medical, Non-fiction, Programming, Written] | deu,eng,fra,go,javascript,jpn,python,spa,sql |
| RTEB(beta) | RTEB Retrieval Embedding Benchmark | 29 | Retrieval: 29 | [Academic, Encyclopaedic, Financial, Legal, Medical, Non-fiction, Programming, Written] | ara,ben,deu,eng,fas,fin,fra,go,hin,ind,javascript,jpn,kor,python,rus,spa,sql,swa,tel,tha,yor,zho |
| RTEB(deu, beta) | RTEB German | 4 | Retrieval: 4 | [Legal, Medical, Non-fiction, Written] | deu |
| RTEB(eng, beta) | RTEB English | 20 | Retrieval: 20 | [Academic, Financial, Legal, Medical, Non-fiction, Programming, Written] | eng,fra,go,javascript,python,spa,sql |
| RTEB(fin, beta) | RTEB Finance | 7 | Retrieval: 7 | [Financial, Non-fiction, Written] | eng |
Expand Down
6 changes: 4 additions & 2 deletions docs/tasks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -889,6 +889,8 @@ The following tables give you an overview of the tasks in MTEB.
| [SwahiliNewsClassification.v2](https://huggingface.co/datasets/Mollel/SwahiliNewsClassification) (Davis et al., 2020) | ['swa'] | Classification | s2s | [News, Written] | None | None |
| [SweFaqRetrieval](https://spraakbanken.gu.se/en/resources/superlim) (Berdi{\v{c, 2023) | ['swe'] | Retrieval | s2s | [Government, Non-fiction, Written] | None | None |
| [SweRecClassification.v2](https://aclanthology.org/2023.nodalida-1.20/) (Nielsen et al., 2023) | ['swe'] | Classification | s2s | [Reviews, Written] | None | None |
| [SwedishPatentCPCGroupClassification](https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-368254) (Salim et al., 2025) | ['swe'] | MultilabelClassification | s2s | [Government, Legal] | None | None |
| [SwedishPatentCPCSubclassClassification](https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-368254) (Salim et al., 2025) | ['swe'] | MultilabelClassification | s2s | [Government, Legal] | None | None |
| [SwedishSentimentClassification.v2](https://huggingface.co/datasets/swedish_reviews) | ['swe'] | Classification | s2s | [Reviews, Written] | None | None |
| [SwednClusteringP2P](https://spraakbanken.gu.se/en/resources/swedn) (Monsen et al., 2021) | ['swe'] | Clustering | p2p | [News, Non-fiction, Written] | None | None |
| [SwednClusteringS2S](https://spraakbanken.gu.se/en/resources/swedn) (Monsen et al., 2021) | ['swe'] | Clustering | s2s | [News, Non-fiction, Written] | None | None |
Expand Down Expand Up @@ -1937,7 +1939,7 @@ The following tables give you an overview of the tasks in MTEB.
| suz | Sunwar | Sino-Tibetan | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| svk | Slovakian Sign Language | Sign Language | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| swa | Swahili (macrolanguage) | Atlantic-Congo | 0 | 1 | 0 | 1 | 7 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 17 |
| swe | Swedish | Indo-European | 0 | 1 | 0 | 6 | 9 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 28 |
| swe | Swedish | Indo-European | 0 | 1 | 0 | 6 | 9 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 1 | 0 | 1 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 30 |
| swg | Swabian | Indo-European | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| swh | Swahili (individual language) | Atlantic-Congo | 0 | 0 | 0 | 3 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 |
| swp | Suau | Austronesian | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
Expand Down Expand Up @@ -2138,7 +2140,7 @@ The following tables give you an overview of the tasks in MTEB.
| zty | Yatee Zapotec | Otomanguean | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| zul | Zulu | Atlantic-Congo | 0 | 0 | 0 | 2 | 3 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 |
| zyp | Zyphe Chin | Sino-Tibetan | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| Total | None | None | None | 0 | 55 | 49 | 1496 | 872 | 321 | 7 | 137 | 22 | 5 | 0 | 3 | 29 | 96 | 4 | 68 | 702 | 91 | 2 | 2 | 6 | 7 | 37 | 24 |
| Total | None | None | None | 0 | 55 | 49 | 1496 | 872 | 321 | 7 | 137 | 22 | 5 | 0 | 3 | 31 | 96 | 4 | 68 | 702 | 91 | 2 | 2 | 6 | 7 | 37 | 24 |

<!-- TASK LANG TABLE END -->

Expand Down
Loading