Added VDR Model

[ayush1298]

This PR is related to addition of VDR model and Training Dataset given in issue #1769

• Model Addition: llamaindex/vdr-2b-multi-v1 model added

Code Quality

• Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

• Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

• New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
• Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding a model checklist

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.

[ayush1298]

@Samoed, can you please review this?
I have some doubts:

1. When I tried to load the model using mteb.get_model(model_name = "llamaindex/vdr-2b-multi-v1" revision = "2c4e54c8db4071cc61fc3c62f4490124e40c37db"), then it gives me the following warning:

WARNING:mteb.model_meta: Loader not specified for model llamaindex/vdr-2b-multi-v1, loading using sentence transformers.

I don't know why it is using sentence transformers and not the Custom one.
2. In code given at hugging face, there is a function name encode_documents, under which it is using following code for preparing inputs:

cache_position = torch.arange(0, len(queries))
inputs = model.prepare_inputs_for_generation(
   **inputs, cache_position=cache_position, use_cache=False)

But, this was giving a following dimension mismatch error to me:

RuntimeError: The size of tensor a (43) must match the size of tensor b (512) at non-singleton dimension 0

The full Traceback error is:

ERROR:mteb.evaluation.MTEB:Error while evaluating MIRACLReranking: The size of tensor a (43) must match the size of tensor b (512) at non-singleton dimension 0
Traceback (most recent call last):
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/model_check.py", line 10, in <module>
    results = evaluation.run(model)
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/MTEB.py", line 673, in run
    raise e
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/MTEB.py", line 626, in run
    results, tick, tock = self._run_eval(
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/MTEB.py", line 308, in _run_eval
    results = task.evaluate(
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/abstasks/AbsTask.py", line 137, in evaluate
    scores[hf_subset] = self._evaluate_subset(
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/tasks/Reranking/multilingual/MIRACLReranking.py", line 98, in _evaluate_subset
    scores = evaluator(model)
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 70, in __call__
    scores = self.compute_metrics(model)
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 75, in compute_metrics
    self.compute_metrics_batched(model)
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 87, in compute_metrics_batched
    model.encode(
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/models/vdr_models.py", line 73, in encode
    return self.get_text_embeddings(sentences, **kwargs)
  File "/Users/ayushmunot/Desktop/OpenSource/mteb/mteb/models/vdr_models.py", line 109, in get_text_embeddings
    output = self.model(
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1700, in forward
    outputs = self.model(
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1096, in forward
    causal_mask = self._update_causal_mask(
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1222, in _update_causal_mask
    causal_mask = self._prepare_4d_causal_attention_mask_with_cache_position(
  File "/Users/ayushmunot/miniconda3/envs/ayushOS/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1301, in _prepare_4d_causal_attention_mask_with_cache_position
    causal_mask *= diagonal_attend_mask
RuntimeError: The size of tensor a (43) must match the size of tensor b (512) at non-singleton dimension 0

So, I change the above piece of code to:

seq_length = inputs["input_ids"].shape[1]
cache_position = torch.arange(0, seq_length).to(self.device)
inputs = self.model.prepare_inputs_for_generation(
    **inputs, cache_position=cache_position, use_cache=False
)

It's running fine after this, can you confirm if this is correct?

3. Where exactly the dataset needs to be added, should I add it under mteb/tasks/Reranking/multilingual with a new file name vdrmultilingual.py

[Samoed]

When I tried to load the model using mteb.get_model(model_name = "llamaindex/vdr-2b-multi-v1" revision = "2c4e54c8db4071cc61fc3c62f4490124e40c37db"), then it gives me the following warning

I think you installed the library incorrectly. It looks like you ran pip install . instead of pip install -e ., which means you're running an mteb version where your VDR model doesn't exist.

Where exactly the dataset needs to be added, should I add it under mteb/tasks/Reranking/multilingual with a new file name vdrmultilingual.py

If you're about training datasets, then you need to add it to model meta like

model = ModelMeta(
    ....,  
    training_datasets={
      #   llamaindex/vdr-multilingual-train
   },
)

[ayush1298]

I think you installed the library incorrectly. It looks like you ran pip install . instead of pip install -e ., which means you're running an mteb version where your VDR model doesn't exist.

Do I need to do pip install -e . every time I make changes in the model file?

Where exactly the dataset needs to be added, should I add it under mteb/tasks/Reranking/multilingual with a new file name vdrmultilingual.py

If you're about training datasets, then you need to add it to the model meta like

model = ModelMeta(
    ....,  
    training_datasets={
      #   llamaindex/vdr-multilingual-train
   },
)

Can you once see the issue #1769, I think it's not only about adding it to the model meta, it's about adding it as a task also.

[Samoed]

Do I need to do pip install -e . every time I make changes in the model file?

No. -e means --editable, so you will run library as now in your code

Can you once see the issue #1769, I think it's not only about adding it to the model meta, it's about adding it as a task also.

We can, but I think you can add model in this PR, and dataset in another

[ayush1298]

We can, but I think you can add the model in this PR, and the dataset in another

Should I add it under mteb/tasks/Reranking/multilingual with a new file name vdrmultilingual.py in another PR?

[Samoed]

Should I add it under mteb/tasks/Reranking/multilingual with a new file name vdrmultilingual.py in another PR?

Yes

[Samoed]

Great work!