Skip to content

fix: Cache language filtering#3612

Merged
KennethEnevoldsen merged 2 commits intomainfrom
fix_cache_filtering
Nov 25, 2025
Merged

fix: Cache language filtering#3612
KennethEnevoldsen merged 2 commits intomainfrom
fix_cache_filtering

Conversation

@Samoed
Copy link
Member

@Samoed Samoed commented Nov 24, 2025

Ref #3611

Previously this code

import mteb
from mteb.cache import ResultCache

cache = ResultCache()
cache.download_from_remote()

b = mteb.get_benchmark("ViDoRe(v3)")
vidore3_full_res = cache.load_results(tasks=b.tasks)
print(vidore3_full_res._get_scores())
new_tasks = []
for t in b.tasks:
    new_t = t.filter_languages(["eng"])
    new_tasks.append(new_t)
vidore3_eng_res = cache.load_results(tasks=new_tasks, validate_and_filter=True)
print(vidore3_eng_res._get_scores())

Would output the same, because filtering result wasn't assigned.

Copy link
Member Author

@Samoed Samoed Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have anything multilingual in mock cache (probably we need to delete most of the results here)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we just add some?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a lot of no_model_name_available or no_revision_available

@KennethEnevoldsen KennethEnevoldsen changed the title fix cache filtering fix: Cache language filtering Nov 25, 2025
@KennethEnevoldsen KennethEnevoldsen merged commit f75bfc4 into main Nov 25, 2025
9 checks passed
@KennethEnevoldsen KennethEnevoldsen deleted the fix_cache_filtering branch November 25, 2025 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants