model: mixedbread-ai/mxbai-edge-colbert-v0-32m and mixedbread-ai/mxbai-edge-colbert-v0-17m

[ayush1298]

Closes #3391
Added both models: https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-17m, https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-32m
If you add a model or a dataset, please add the corresponding checklist:

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision) and
  • mteb.get_model_meta(model_name, revision)
• I have tested the implementation works on a representative set of tasks.
• The model is public, i.e., is available either as an API or the weights are publicly available to download

[Copilot]

Pull request overview

This PR adds two new late-interaction ColBERT models from mixedbread-ai to the MTEB model registry: mxbai-edge-colbert-v0-17m and mxbai-edge-colbert-v0-32m. These are small, efficient retrieval models with 17M and 32M parameters respectively.

Changes:

• Added two new ModelMeta definitions for the mxbai-edge-colbert-v0-17m and mxbai-edge-colbert-v0-32m models with complete metadata including training datasets, citation, and model specifications
• Updated import in mdbr_models.py to use mixedbread_ai_models module for consistency

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
mteb/models/model_implementations/mixedbread_ai_models.py	Adds import for MultiVectorModel and defines two new late-interaction ColBERT model metadata objects with comprehensive details including training datasets, citations, and model specifications
mteb/models/model_implementations/mdbr_models.py	Updates import statement to reference mixedbread_ai_models instead of the older module path for better organization

Comments suppressed due to low confidence (2)

mteb/models/model_implementations/mixedbread_ai_models.py:273

• The commented-out training dataset 'FineWeb' is missing quotes. For consistency with other commented datasets on lines 272, 274, and 275, it should be written as # "FineWeb", if it represents a string value. This would prevent potential errors if the line is uncommented in the future.
  mteb/models/model_implementations/mixedbread_ai_models.py:318
• The commented-out training dataset 'FineWeb' is missing quotes. For consistency with other commented datasets on lines 317, 319, and 320, it should be written as # "FineWeb", if it represents a string value. This would prevent potential errors if the line is uncommented in the future.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ayush1298]

@Samoed On hf they have given results on NanoBEIR and LongEmbed benchmarks, But when I am trying to run below code, I am getting error:

import mteb
model_name1 = "mixedbread-ai/mxbai-edge-colbert-v0-32m"
revision1 = "2f12870a85dae80680b9babc59992c9a2bc59e4a"
model1 = mteb.get_model_meta(model_name1, revision1)

benchmark = mteb.get_benchmark("LongEmbed")
tasks = benchmark.tasks
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model1)

TypeError: RetrievalEvaluator expects a SearchInterface, Encoder, or CrossEncoder, got <class 'mteb.models.model_meta.ModelMeta'>

[Samoed]

You should change mteb.get_model_meta to mteb.get_model and it's better to use mteb.evaluate

[ayush1298]

You should change mteb.get_model_meta to mteb.get_model and it's better to use mteb.evaluate

sorry I was testing and mistakenly used mteb.get_model_meta instead of mteb.get_model.

[ayush1298]

1 more doubt, when I am doing evaluations like these:

import mteb
model_name1 = "mixedbread-ai/mxbai-edge-colbert-v0-32m"
revision1 = "2f12870a85dae80680b9babc59992c9a2bc59e4a"
model1 = mteb.get_model(model_name1)
benchmark2 = mteb.get_benchmark("NanoBEIR")
tasks2 = benchmark2.tasks
results2 = mteb.evaluate(model1, tasks=tasks2)

No JSON files are getting created for individual tasks in the benchmark. Is it that I have to see results in the following way only:

task_result2 = results2.task_results
for t in task_result2:
    print(f"Task name is: {t.task_name} and results is : {t.scores} ")

Also, do we also report some kind of average results of each metric across all tasks for a benchmark?

[Samoed]

1. You can just do

benchmark2 = mteb.get_benchmark("NanoBEIR")
results2 = mteb.evaluate(model1, benchmark2)

2. No JSON files are getting created

Are you sure that results are not in ~/.cache/mteb/?

[ayush1298]

1. You can just do

benchmark2 = mteb.get_benchmark("NanoBEIR")
results2 = mteb.evaluate(model1, benchmark2)

2. No JSON files are getting created

Are you sure that results are not in ~/.cache/mteb/?

After running, results folder get created, but under there was just model_meta.json, not others.
Also, why it would be under ~/.cache/mteb/, and not in results folder that gets created after mteb.evaluate?

[Samoed]

I run this with baseline/random-encoder-baseline (I didn't want to recreate venv for pylate, because I'm using 3.13)

import mteb
model_name1 = "baseline/random-encoder-baseline"
revision1 = "2f12870a85dae80680b9babc59992c9a2bc59e4a"
model1 = mteb.get_model(model_name1)
benchmark2 = mteb.get_benchmark("NanoBEIR")
results2 = mteb.evaluate(model1, benchmark2)

And my I can see all task results in ~/.cache/mteb/results/baseline__random-encoder-baseline/1.

You can also specify your folder with

cache = mteb.cache.ResultCache("/path/to/folder")
mteb.evaluate(..., cache=cache)

Also, why it would be under ~/.cache/mteb/, and not in results folder that gets created after mteb.evaluate?

This change from v2. Now we store all results in cache, but you can specify any directory for that

[ayush1298]

@Samoed Got it. This works. However, I am unable to run the evaluation on the actual model due to some dependency issues.

[Samoed]

I evaluated without problems on NanoBeir

[ayush1298]

I evaluated without problems on NanoBeir

Can you check whether results are matching for NanoBEIR, with results given at: https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-17m. Its possible that there may be some problem from my side as I was running it on google colab.

[Samoed]

I've got same results

[ayush1298]

I've got same results

Could you also add them in results repo?