Adding results for nvidia_llama_nemoretriever_colembed models

[bschifferer]

Checklist

• My model has a model sheet, report or similar
• My model has a reference implementation in
  • No, but there is an existing PR Adding nvidia/llama-nemoretriever-colembed models mteb#2861
• The results submitted is obtained using the reference implementation (script is provided in the HuggingFace checkpoints)
• My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
• solemnly swear that for all results submitted I have not on the evaluation dataset including training splits. If I have I have disclosed it clearly.

Models are available on HF (will be publish soon)
https://huggingface.co/nvidia/llama-nemoretriever-colembed-1b-v1
https://huggingface.co/nvidia/llama-nemoretriever-colembed-3b-v1

They contain a description of the training datasets and a script to run MTEB VisualDocumentRetrieval

[bschifferer]

@KennethEnevoldsen , @Samoed , @isaac-chung - Hello, can you might help me what the error is?

KeyError: 'intfloat/multilingual-e5-large'

I dont change anything related to intfloat model. Why is there a key error? Can I somehow trigger to rerun the tests?

[Samoed]

I think you've just run a task that intfloat/multilingual-e5-large haven't run. I fixed this issue in #229

[bschifferer]

@Samoed - How did I run the task - I just open a PR to submit our results.

Do I ned to rebase/change something in my PR?

[Samoed]

No, you don't need to anything. This is fine that this CI failed

[rnyak]

@Samoed hello. any ETA for merging your PR ? thanks.

[Samoed]

This PR is not blocking merging your results

[isaac-chung]

The model PR only has minor comments. This is good to go.

[bschifferer]

Thanks a lot for the quick feedback and review. I thought the checks are mandatory to pass, but it seems it is optional.

@isaac-chung @Samoed : Do you think it is possible to merge the PR by Monday afternoon? We want to share/promote our models, but we are waiting until the models are available on the leaderboards of MTEB VisualDocumentRetrieval and ViDoRe V1+V2 (both uses the results from the MTEB repository, now). That would be great

I wish you a great weekend.

[Samoed]

I think it will be possible, but you should a bit fix up your model metadata

[bschifferer]

I think it will be possible, but you should a bit fix up your model metadata

Thanks @Samoed - I just updated my PR in the MTEB library for model_meta_data and updated the .json files in this PR