task: add FIVR fine-grained video retrieval (v2v) - #5134
Conversation
| "playwright>=1.49.0", | ||
| ] | ||
| video = ["torchcodec"] | ||
| fivr = ["pytubefix>=10.11.0", "yt-dlp>=2026.7.4"] |
There was a problem hiding this comment.
Tasks should be run without additional dependencies
There was a problem hiding this comment.
Ok I’ll remove these FIVR-specific downloader dependencies from pyproject.toml and uv.lock, because downloading the videos will be handled only by the standalone preparation script. I’ll also remove the downloader imports from the MTEB task itself, so the task will only read videos from a prepared local directory and report clearly if any required files are missing.
| return target | ||
|
|
||
|
|
||
| def _download_with_ytdlp(video_id: str, video_dir: Path) -> Path: |
There was a problem hiding this comment.
This is too much downloading for tasks
There was a problem hiding this comment.
Agree, I’ll move all downloading to the standalone preparation script and make the task use a prepared local video directory instead.
There was a problem hiding this comment.
Pull request overview
Adds the FIVR-5K “fine-grained incident video retrieval” benchmark to MTEB as three video-to-video (Any2AnyRetrieval, v2v) tasks, backed by a frozen metadata-only Hugging Face dataset and supporting scripts for reproducible construction/auditing.
Changes:
- Introduces
FIVR5KDSVRRetrieval,FIVR5KCSVRRetrieval, andFIVR5KISVRRetrievaltasks loading a frozen metadata dataset and requiring a prebuilt local video directory. - Adds data tooling to (a) audit YouTube availability, (b) construct/push the metadata dataset, and (c) run a shared-corpus evaluation encoding the corpus once across all three regimes.
- Adds descriptive statistics JSON artifacts for the three new tasks.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/data/fivr/run_shared_evaluation.py | Shared-corpus evaluation runner for the 3 regimes with embedding shard checkpointing. |
| scripts/data/fivr/dataset_card.md | Dataset card template for publishing the metadata-only HF dataset. |
| scripts/data/fivr/create_data.py | Construction script for metadata freeze + optional media materialization + hub push. |
| scripts/data/fivr/audit_availability.py | Audits VideoEval’s frozen manifest availability without downloading media. |
| mteb/tasks/retrieval/zxx/fivr_5k_retrieval.py | New FIVR-5K retrieval tasks with local-video resolution and HF metadata loading. |
| mteb/tasks/retrieval/zxx/init.py | Exports the new FIVR task classes. |
| mteb/descriptive_stats/Image/Any2AnyRetrieval/FIVR5KISVRRetrieval.json | Descriptive stats for ISVR regime. |
| mteb/descriptive_stats/Image/Any2AnyRetrieval/FIVR5KDSVRRetrieval.json | Descriptive stats for DSVR regime. |
| mteb/descriptive_stats/Image/Any2AnyRetrieval/FIVR5KCSVRRetrieval.json | Descriptive stats for CSVR regime. |
Suppressed comments (1)
scripts/data/fivr/audit_availability.py:80
- After making
yt_dlpan optional import,_probe_ytdlpshould fail gracefully (and with a clear error) when the backend isyt-dlpbut the package is not installed, instead of raising an AttributeError onyt_dlp.YoutubeDL.
if request_delay:
time.sleep(request_delay)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| import yt_dlp # type: ignore[import-untyped] |
| try: | ||
| from pytubefix import YouTube # type: ignore[import-untyped] |
Summary
Adds FIVR-5K as three video-to-video
Any2AnyRetrievaltasks using FIVR-200K (paper) annotations and VideoEval's frozen FIVR-5K manifest:FIVR5KDSVRRetrieval: ND + DSFIVR5KCSVRRetrieval: ND + DS + CSFIVR5KISVRRetrieval: ND + DS + CS + ISAll use full-ranking
map_at_3188, equivalent to the source mAP metric.Dataset
The availability freeze checked 3,445 unique manifest IDs: 3,217 available and 228 unavailable/restricted. The final benchmark has 29 queries and 3,188 corpus videos, with 417/485/700 DSVR/CSVR/ISVR qrels. Two unavailable query videos were dropped; per-query and per-category losses are recorded.
Metadata: Cerru02/FIVR-5K-MTEB at
27278cba0b40e0dc764c652efb2e9eed9b997c83.Only metadata, IDs, source URLs, audits, and qrels are hosted. FIVR/ViSiL/VideoEval metadata is Apache-2.0; this does not grant redistribution rights to the underlying YouTube media, so no video bytes are hosted.
Results (full mAP)
mteb/baseline-random-encoderQwen/Qwen3-VL-Embedding-2BQwen used the current surviving corpus and MTEB 16-frame preprocessing, so these are not direct reproductions of VideoEval's reported results.
Validation
make lint,make typecheckmake test: 6,598 passed, 240 skippedCloses #5128
Related to #4842