Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
236 commits
Select commit Hold shift + click to select a range
9556f99
misc: Add image classification descriptive stats implementation (#2045)
isaac-chung Feb 13, 2025
fadba48
Update tasks table
github-actions[bot] Feb 13, 2025
01fd6fb
fix: Add column descriptions to leaderboard (#2039)
KennethEnevoldsen Feb 13, 2025
3537223
fix: Add BRIGHT (long) and fix bug in TaskResult.filter_and_validate(…
KennethEnevoldsen Feb 13, 2025
68ff565
1.34.12
invalid-email-address Feb 13, 2025
eb32719
misc: Add image clustering descriptive stats implementation (#2057)
isaac-chung Feb 13, 2025
50b8e7b
fix: Update embed_dim for jina models (#2058)
KennethEnevoldsen Feb 13, 2025
48ef6f4
Update tasks table
github-actions[bot] Feb 13, 2025
8b7f2f8
1.34.13
invalid-email-address Feb 13, 2025
02d2583
Add giga embeddings (#1741)
Samoed Feb 13, 2025
20df284
misc: Add ZS and multilabel image classification descriptive stats im…
isaac-chung Feb 14, 2025
e090330
Update tasks table
github-actions[bot] Feb 14, 2025
bef4046
Rename MIEB task classes with duplicated names (#2061)
Samoed Feb 14, 2025
3cf7b15
misc: Add VisualSTS descriptive stats (#2062)
isaac-chung Feb 14, 2025
479fa20
Update tasks table
github-actions[bot] Feb 14, 2025
76e05dd
fix: Added gte models (#1539)
KennethEnevoldsen Feb 14, 2025
8604e07
fix: Add climate fever v2 (#1873)
mina-parham Feb 14, 2025
11ced79
Update tasks table
github-actions[bot] Feb 14, 2025
c6829d3
fix: Updating paper scripts (#1958)
KennethEnevoldsen Feb 14, 2025
26708c5
1.34.14
invalid-email-address Feb 14, 2025
5f4b593
Add datasets for a benchmark newly introduced for "Engineering" domai…
mehrzadshm Feb 15, 2025
dbda3c5
Update tasks table
github-actions[bot] Feb 15, 2025
50cc1c9
misc: update model names to adjust for adding to results repo (#2074)
isaac-chung Feb 16, 2025
04c9993
misc: Add all image classification descriptive stats (#2073)
isaac-chung Feb 17, 2025
3dbdeb1
Update tasks table
github-actions[bot] Feb 17, 2025
efaa990
ci: Rerun tests that fail due to networking issues. (#2029)
sam-hey Feb 17, 2025
26360a0
fix: generate metadata (#2063)
sam-hey Feb 17, 2025
8d4adbf
1.34.15
invalid-email-address Feb 17, 2025
efe2578
fix: add missing `e5` training datasets (#2065)
Samoed Feb 17, 2025
8ef26d0
1.34.16
invalid-email-address Feb 17, 2025
b14963f
fix: Ensure voyage model uses different naming scheme (#2083)
KennethEnevoldsen Feb 17, 2025
2d1f10d
1.34.17
invalid-email-address Feb 17, 2025
07562f4
fix: Freeze model/rank columns in leaderboard (#2044)
shikhar1729 Feb 17, 2025
879b243
1.34.18
invalid-email-address Feb 17, 2025
12d9b96
fix: Fixed previous incorrect specification of splits for CMTEB ( MTE…
KennethEnevoldsen Feb 17, 2025
72d454f
1.34.19
invalid-email-address Feb 17, 2025
c6e5123
Remove duplicated string in docstring of TaskMetadata class (#2087)
dantetemplar Feb 17, 2025
1006770
fix: Smarter leaderboard caching with cachetools (#2085)
x-tabdeveloping Feb 17, 2025
6637ff9
fix: Missing fixes for #2086 - change MultilingualSentiment split fro…
KennethEnevoldsen Feb 17, 2025
1f9cfc8
1.34.20
invalid-email-address Feb 17, 2025
1b1d327
merge gme models (#2089)
Samoed Feb 17, 2025
3deb7ea
fix: Add back task filtering by modalities (#2080)
isaac-chung Feb 18, 2025
544bcd1
1.34.21
invalid-email-address Feb 18, 2025
bbfbc45
Added gtr-t5-base/large/xl/xxl metadata to mteb (#2092)
sufen-f Feb 18, 2025
0371102
misc: Add Any2TextMutipleChoice Descriptive Statistics (#2095)
isaac-chung Feb 18, 2025
9ca55f0
Update tasks table
github-actions[bot] Feb 18, 2025
e0b364b
fix: Updated model annotations for GTE, e5, gritlm, and SFR models (#…
KennethEnevoldsen Feb 19, 2025
6b9f945
fix: Update links (#2098)
Muennighoff Feb 19, 2025
06489ab
1.34.22
invalid-email-address Feb 19, 2025
c69b8c3
Add model inf-retriever-v1-1.5b (#2106)
SamuelYang1 Feb 20, 2025
caa0b77
docs: Fix typos & refine text (#2102)
Muennighoff Feb 20, 2025
56a7b1a
misc: Run Zeroshot Classification Descriptive Stats (#2105)
isaac-chung Feb 20, 2025
6e0c87a
Update tasks table
github-actions[bot] Feb 20, 2025
6a71485
fix: add warning about task category conversion (#2108)
isaac-chung Feb 20, 2025
c91fbd0
1.34.23
invalid-email-address Feb 20, 2025
c052bbb
fix: Add codesage-large-v2 (#2090)
Aradhye2002 Feb 20, 2025
226b652
1.34.24
invalid-email-address Feb 20, 2025
cb42f4a
fix: add training data to BGE-m3-custom-fr (#2110)
KennethEnevoldsen Feb 20, 2025
dbe7559
1.34.25
invalid-email-address Feb 20, 2025
fb14e0c
fix: Upgrade ruff to be gradio compatible (#2111)
KennethEnevoldsen Feb 20, 2025
7538a2d
1.34.26
invalid-email-address Feb 20, 2025
276840f
docs: Follow google docstring format (#2115)
KennethEnevoldsen Feb 20, 2025
f3e4a9a
Update leaderboard_refresh.yaml (#2121)
Samoed Feb 21, 2025
463ca54
fix InstructSentenceTransformer Model name (#2125)
Samoed Feb 21, 2025
b032f98
fix voyage (#2127)
Samoed Feb 21, 2025
44cfa9b
fix: update e5 instruct training data (#2129)
Samoed Feb 21, 2025
d5a40e6
1.34.27
invalid-email-address Feb 21, 2025
950e3ab
format
KennethEnevoldsen Feb 21, 2025
de2e3e3
Update tasks table
github-actions[bot] Feb 21, 2025
e7735b2
fix: Add 2 new Static Sentence Transformer models (#2112)
tomaarsen Feb 21, 2025
2874e0c
1.34.28
invalid-email-address Feb 21, 2025
e6eb473
add is_cross_encoder (#1869)
Samoed Feb 21, 2025
17a120a
Qodo embed 1 1.5 b (#2137)
talshef Feb 23, 2025
4389501
misc: merge summary retrieval into bitext mining (#2140)
isaac-chung Feb 24, 2025
0163342
test: fix dataset availability test (#2141)
KennethEnevoldsen Feb 24, 2025
760fcaf
fix: Update NVIDIA-Embed training data (#2143)
KennethEnevoldsen Feb 24, 2025
9f6cc4e
1.34.29
invalid-email-address Feb 24, 2025
8538e93
fix: Add annotations for Voyage exp (#2144)
KennethEnevoldsen Feb 24, 2025
25cd62d
1.34.30
invalid-email-address Feb 24, 2025
8e97d36
Fix tokens num in cde models (#2148)
Samoed Feb 24, 2025
0e624b2
feat: Add Qodo-Embed-1-7B model metadata and rename existing model (#…
talshef Feb 24, 2025
4d23c6c
1.35.0
invalid-email-address Feb 24, 2025
bd2a67c
misc: add Any2AnyRetrievalDescriptiveStatistics (#2139)
isaac-chung Feb 24, 2025
ef3f4f0
Update tasks table
github-actions[bot] Feb 24, 2025
a7dc95a
Added zero-shot percentages and different filtering scheme (#2153)
x-tabdeveloping Feb 25, 2025
565e29c
fix: Incorrect annotations for Mistral-based embedding models (#2157)
KennethEnevoldsen Feb 25, 2025
90ec21c
1.35.1
invalid-email-address Feb 25, 2025
8afb78a
Update FaMTEBRetrieval.py (#2171)
garciasces Feb 26, 2025
331cded
Update tasks table
github-actions[bot] Feb 26, 2025
6cc1822
fix: Add Training data annotations (#2173)
KennethEnevoldsen Feb 26, 2025
ed0cb31
1.35.2
invalid-email-address Feb 26, 2025
dea231b
feat: Add MIEB and MIEB-lite as benchmarks (#2035)
isaac-chung Feb 27, 2025
dbcbf54
Update tasks table
github-actions[bot] Feb 27, 2025
afe1739
1.36.0
invalid-email-address Feb 27, 2025
62b33f2
fix: update training datasets and revision for jina models (#2179)
Feb 27, 2025
1959c73
fix: Add more training data annotations (#2178)
KennethEnevoldsen Feb 27, 2025
4a0bb5c
1.36.1
invalid-email-address Feb 27, 2025
43d15f1
Added training data annotation for e5-base-4k (#2186)
x-tabdeveloping Feb 28, 2025
1b23d4e
fix: Added training data annotations to MXBAI (#2185)
x-tabdeveloping Feb 28, 2025
7daf893
fix: Update MTEB(Scandinavian) to use new DanFEVER (#2180)
KennethEnevoldsen Feb 28, 2025
0307102
fix: Added training data annotation for MMLW models (#2188)
x-tabdeveloping Feb 28, 2025
7642c07
1.36.2
invalid-email-address Feb 28, 2025
0901cf6
fix: Added training data for sentence-croissant (#2189)
x-tabdeveloping Feb 28, 2025
d4b691f
1.36.3
invalid-email-address Feb 28, 2025
3325f7e
fix: update ru models annotation (#2181)
Samoed Feb 28, 2025
c04d158
1.36.4
invalid-email-address Feb 28, 2025
fee6fc0
fix: Alphabetical ordering of tasks in dropdowns (#2191)
ayush1298 Feb 28, 2025
0631089
1.36.5
invalid-email-address Feb 28, 2025
7345235
misc: Speed up qrel creation in any2anyretrieval (#2196)
isaac-chung Feb 28, 2025
29464ac
use 'mteb.MTEB' instead of 'MTEB' for custom model (#2199)
yaya-sy Feb 28, 2025
1c8d715
add base models for e5 (#2183)
Samoed Mar 2, 2025
7af37d4
add similar datasets (#2205)
Samoed Mar 2, 2025
587892d
add labse annotation (#2182)
Samoed Mar 2, 2025
761a174
fix: Fixed leaderboard crash (#2221)
x-tabdeveloping Mar 3, 2025
e57cd50
1.36.6
invalid-email-address Mar 3, 2025
2dd1391
fix: More training data annotations (#2220)
x-tabdeveloping Mar 3, 2025
546e0c4
1.36.7
invalid-email-address Mar 3, 2025
4ee4e7c
Add LLM2CLIP (OpenAI variants) (#2222)
isaac-chung Mar 3, 2025
c5fded2
Change `dataset on HF` test to use official api (#2213)
Samoed Mar 3, 2025
3e991bd
Descriptive stats functions for Any2AnyMC and ImageTextPC (#2197)
imenelydiaker Mar 3, 2025
cc47225
Update tasks table
github-actions[bot] Mar 3, 2025
ee514cb
fix: Add training data annotations to uderver-bloom models (#2210)
KennethEnevoldsen Mar 3, 2025
4de58c3
1.36.8
invalid-email-address Mar 3, 2025
a87927b
Add comment to `voyage-3-m-exp` model (#2229)
Samoed Mar 3, 2025
3a9d271
docs: Update description of EURLex (#2231)
KennethEnevoldsen Mar 4, 2025
7f7d3e8
Automatically add similar tasks to training_tasks (#2228)
Samoed Mar 4, 2025
6129282
Remove overlapping legends from radar chart (#2195)
ayush1298 Mar 5, 2025
40b89db
misc: Run Any2AnyRetrieval descriptive stats (#2223)
isaac-chung Mar 6, 2025
e81d109
Update tasks table
github-actions[bot] Mar 6, 2025
43cb205
misc: Add rest of the vision centric and compositionality descriptive…
isaac-chung Mar 6, 2025
d8e73e7
Update tasks table
github-actions[bot] Mar 6, 2025
a4456ec
Fix `calculate_memory_usage_mb` in adding_a_model.md (#2271)
Samoed Mar 6, 2025
f964829
Add Arabic-Triplet-Matryoshka-V2 model metadata to MTEB (#2270)
omarnj-lab Mar 7, 2025
9d6e1a9
fix: Add WebFAQ Retrieval dataset (#2236)
michaeldinzinger Mar 7, 2025
a67c4d0
Update tasks table
github-actions[bot] Mar 7, 2025
1841aca
1.36.9
invalid-email-address Mar 7, 2025
c456111
fix: Formatting issue in Performance Plot (#2237)
ayush1298 Mar 7, 2025
1d41474
1.36.10
invalid-email-address Mar 7, 2025
55b9a0e
ci: run test_dataset_on_hf separately (#2201)
sam-hey Mar 7, 2025
fb1b04c
add gemini-embedding-exp-03-07 (#2279)
jhyuklee Mar 7, 2025
9513f15
update link (#2281)
jhyuklee Mar 7, 2025
e628bce
fix: Run remaining MIEB desc stats (#2288)
isaac-chung Mar 8, 2025
dd7008d
Update tasks table
github-actions[bot] Mar 8, 2025
18ed1bb
1.36.11
invalid-email-address Mar 8, 2025
f840f7d
fix: Added Filter Modality (#2262)
ayush1298 Mar 9, 2025
6284f25
1.36.12
invalid-email-address Mar 9, 2025
5dce601
fix: Add `ModelMeta` license & custom validations (#2293)
Samoed Mar 9, 2025
02003b1
1.36.13
invalid-email-address Mar 9, 2025
5b30d84
ci: Add pre-commit hook (#2194)
sam-hey Mar 10, 2025
5e3915e
Update tasks table
github-actions[bot] Mar 10, 2025
6193db1
fix: bug in voyage implementation (#2304)
KennethEnevoldsen Mar 10, 2025
c4d2888
1.36.14
invalid-email-address Mar 10, 2025
746b411
fix: Update voyage name to include Org. (#2322)
KennethEnevoldsen Mar 11, 2025
5f6872e
1.36.15
invalid-email-address Mar 11, 2025
7965aad
Added VDR Model (#2290)
ayush1298 Mar 11, 2025
8f6bf45
fix: Resolve conflicting dependencies (#2323)
KennethEnevoldsen Mar 11, 2025
122eaa1
1.36.16
invalid-email-address Mar 11, 2025
fc176ad
fix: remove SyntaxWarnings in py312 (#2325)
KennethEnevoldsen Mar 11, 2025
8b14281
1.36.17
invalid-email-address Mar 11, 2025
034da4d
fix: add annotation models for stella zh (#2277)
KennethEnevoldsen Mar 11, 2025
d58f229
1.36.18
invalid-email-address Mar 11, 2025
ae83b5f
fix: Add ModelMeta rubert-mini-frida, BERTA (#2330)
sergeyz-zh Mar 11, 2025
849efbb
docs: fix typos
Muennighoff Mar 11, 2025
f16b3f9
1.36.19
invalid-email-address Mar 11, 2025
04cfe4d
fix: Add WebFAQ bitext mining tasks (#2326)
michaeldinzinger Mar 12, 2025
d716408
Update tasks table
github-actions[bot] Mar 12, 2025
c40747f
1.36.20
invalid-email-address Mar 12, 2025
fc329ba
fix: Add `trust_remote_code` to MIRACLRetrieval
KennethEnevoldsen Mar 13, 2025
2d45653
fix: Add `trust_remote_code` to MIRACLRetrieval (#2344)
KennethEnevoldsen Mar 13, 2025
b4f8f60
1.36.21
invalid-email-address Mar 13, 2025
d6d8552
fix: Correctly pass trust remote code to Miracl
KennethEnevoldsen Mar 13, 2025
7576b57
Merge branch 'fix-miracl'
KennethEnevoldsen Mar 13, 2025
75961a0
fix: Ensure MIRACL pass trust_remote_code (#2346)
KennethEnevoldsen Mar 13, 2025
eadc38d
Merge branch 'main' of https://github.com/embeddings-benchmark/mteb
KennethEnevoldsen Mar 13, 2025
983e7fa
1.36.22
invalid-email-address Mar 13, 2025
efeb1c9
add-Data Korean Clustering dataset (KLUE-modified) (#2283)
OnAnd0n Mar 15, 2025
e34b139
Rename dunzhang and Jasper models to NovaResearch (#2373)
ayush1298 Mar 15, 2025
773211b
correct MIEB dataset stats (#2374)
gowitheflow-1998 Mar 15, 2025
f31dc32
Update tasks table
github-actions[bot] Mar 15, 2025
6fddbb0
Correct -1 to No information in Zero shot (#2381)
ayush1298 Mar 17, 2025
1bd844f
fix leaderboard (#2385)
KennethEnevoldsen Mar 17, 2025
99eb94b
fix: Reduce logging and Warnings (#2349)
ayush1298 Mar 17, 2025
29dba73
1.36.23
invalid-email-address Mar 17, 2025
60c0a75
fix: b1ade (#2386)
KennethEnevoldsen Mar 17, 2025
a0f3583
1.36.24
invalid-email-address Mar 17, 2025
43b5b69
fix: pin gradio dependency to ensure leaderboards works (#2387)
KennethEnevoldsen Mar 17, 2025
9ec55f7
1.36.25
invalid-email-address Mar 17, 2025
cf26764
fix: Ensure BrightRetrieval is valid to run (#2334)
KennethEnevoldsen Mar 18, 2025
042d6e7
Update tasks table
github-actions[bot] Mar 18, 2025
b3a9191
1.36.26
invalid-email-address Mar 18, 2025
5ebee24
Pass task name to all evaluators (#2389)
Samoed Mar 18, 2025
e7b04a6
fix: renaming Zeroshot -> ZeroShot (#2395)
KennethEnevoldsen Mar 20, 2025
349d5a8
1.36.27
invalid-email-address Mar 20, 2025
cf84a79
fix: Update AmazonPolarityClassification license (#2402)
KennethEnevoldsen Mar 20, 2025
a0990cb
fix b1ade name (#2403)
Samoed Mar 20, 2025
d2dc2f6
1.36.28
invalid-email-address Mar 20, 2025
8be95b7
Minor style changes (#2396)
KennethEnevoldsen Mar 21, 2025
e2476d2
Added new dataset and tasks - ClusTREC-covid , clustering of thematic…
katzurik Mar 21, 2025
5b0bd56
Update tasks table
github-actions[bot] Mar 21, 2025
cae1575
fix: Major updates to docs + make mieb dep optional (#2397)
KennethEnevoldsen Mar 22, 2025
9c459a8
1.36.29
invalid-email-address Mar 22, 2025
811dbf6
remove Arabic_Triplet_Matryoshka_V2.py (#2405)
Samoed Mar 22, 2025
146a893
Min torchvision>0.2.1 (#2410)
isaac-chung Mar 22, 2025
095851f
fix: Add validation to model_name in `ModelMeta` (#2404)
Samoed Mar 22, 2025
a934610
1.36.30
invalid-email-address Mar 22, 2025
2833138
[MIEB] "capability measured"-Abstask 1-1 matching refactor [1/3]: rei…
gowitheflow-1998 Mar 22, 2025
065159d
Update tasks table
github-actions[bot] Mar 22, 2025
e8faf3f
fix: Add option to remove benchmark from leaderboard (#2417)
KennethEnevoldsen Mar 23, 2025
a25dadb
1.36.31
invalid-email-address Mar 23, 2025
9d9b0b4
fix: Add VDR Multilingual Dataset (#2408)
ayush1298 Mar 23, 2025
34edcd5
Update tasks table
github-actions[bot] Mar 23, 2025
0cdf2e0
1.36.32
invalid-email-address Mar 23, 2025
071741d
HOTFIX: pin setuptools (#2423)
Samoed Mar 24, 2025
39cee62
add __init__.py Clustering > kor folder, And edit __init__.py in C…
OnAnd0n Mar 25, 2025
55c542b
Update tasks table
github-actions[bot] Mar 25, 2025
731c4fc
Update speed dependencies with new setuptools release (#2429)
Samoed Mar 25, 2025
98ab0ef
add richinfoai models (#2427)
richinfo-ai Mar 25, 2025
d3eab6f
Added Memory Usage column on leaderboard (#2428)
ayush1298 Mar 25, 2025
0db0a20
docs: typos; Standardize spacing; Chronological order (#2436)
Muennighoff Mar 26, 2025
8a024be
fix: Add model specific dependencies in pyproject.toml (#2424)
ayush1298 Mar 26, 2025
6ae420d
1.36.33
invalid-email-address Mar 26, 2025
65446e5
[MIEB] "capability measured"-Abstask 1-1 matching refactor [2/3]: rei…
gowitheflow-1998 Mar 26, 2025
19dc625
Update tasks table
github-actions[bot] Mar 26, 2025
dadafbe
Error while evaluating MIRACLRetrievalHardNegatives: 'trust_remote_co…
KennethEnevoldsen Mar 27, 2025
43adb0c
Feat/searchmap preview (#2420)
Free-tek Mar 28, 2025
5af5547
Add Background Gradients in Summary and Task Table (#2392)
ayush1298 Mar 29, 2025
61d3c6c
add ops_moa_models (#2439)
ahxgw Mar 29, 2025
35a8a5b
leaderboard fix (#2456)
ayush1298 Mar 29, 2025
d11934f
ci: cache `~/.cache/huggingface` (#2464)
sam-hey Mar 31, 2025
8799126
[MIEB] "capability measured"-Abstask 1-1 matching refactor [3/3]: rei…
gowitheflow-1998 Apr 1, 2025
5b567bf
Update tasks table
github-actions[bot] Apr 1, 2025
f293d8b
fix: Adds family of NeuML/pubmedbert-base-embedding models (#2443)
nadshe Apr 1, 2025
c617598
fix: add nb_sbert model (#2339)
theatollersrud Apr 1, 2025
42068c6
1.36.34
invalid-email-address Apr 1, 2025
29c7b86
Merge remote-tracking branch 'origin/main' into isaac/merge-maeb-main
isaac-chung Apr 1, 2025
df2dfb0
fix test
isaac-chung Apr 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/dataset_loading.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ jobs:

- name: Install dependencies
run: |
make install-for-tests
make install-for-tests

- name: Run dataset loading tests
run: |
make dataset-load-test
7 changes: 7 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Cache Hugging Face
id: cache-hf
uses: actions/cache@v4
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-hf

- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
install:
@echo "--- 🚀 Installing project dependencies ---"
pip install -e ".[dev]"
pip install -e ".[dev,image]"
pre-commit install

install-for-tests:
@echo "--- 🚀 Installing project dependencies for test ---"
@echo "This ensures that the project is not installed in editable mode"
pip install ".[dev,speedtask]"
pip install ".[dev,image]"

lint:
@echo "--- 🧹 Running linters ---"
Expand Down Expand Up @@ -42,7 +42,7 @@ build-docs:

model-load-test:
@echo "--- 🚀 Running model load test ---"
pip install ".[dev, speedtask, pylate,gritlm,xformers,model2vec]"
pip install ".[dev, pylate,gritlm,xformers,model2vec]"
python scripts/extract_model_names.py $(BASE_BRANCH) --return_one_model_name_per_file
python tests/test_models/model_loading.py --model_name_file scripts/model_names.txt

Expand Down
Loading
Loading