Skip to content

Conversation

@roipony
Copy link
Contributor

@roipony roipony commented Aug 14, 2025

Checklist

  • My model has a model sheet, report or similar
  • [] My model has a reference implementation in mteb/models/ this can be as an API. Instruction on how to add a model can be found here
  • The results submitted is obtained using the reference implementation
  • My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
  • I solemnly swear that for all results submitted I have not on the evaluation dataset including training splits. If I have I have disclosed it clearly.

@roipony roipony changed the title Add files via upload Add granite-vision-embedding results Aug 14, 2025
@github-actions
Copy link

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: ibm-granite/granite-vision-3.3-2b-embedding
Tasks: Vidore2BioMedicalLecturesRetrieval, Vidore2ESGReportsHLRetrieval, Vidore2ESGReportsRetrieval, Vidore2EconomicsReportsRetrieval, VidoreArxivQARetrieval, VidoreDocVQARetrieval, VidoreInfoVQARetrieval, VidoreShiftProjectRetrieval, VidoreSyntheticDocQAAIRetrieval, VidoreSyntheticDocQAEnergyRetrieval, VidoreSyntheticDocQAGovernmentReportsRetrieval, VidoreSyntheticDocQAHealthcareIndustryRetrieval, VidoreTabfquadRetrieval, VidoreTatdqaRetrieval

Results for ibm-granite/granite-vision-3.3-2b-embedding

task_name ibm-granite/granite-vision-3.3-2b-embedding Max result
Vidore2BioMedicalLecturesRetrieval 0.56 0.63
Vidore2ESGReportsHLRetrieval 0.65 0.76
Vidore2ESGReportsRetrieval 0.56 0.57
Vidore2EconomicsReportsRetrieval 0.51 0.58
VidoreArxivQARetrieval 0.84 0.89
VidoreDocVQARetrieval 0.55 0.66
VidoreInfoVQARetrieval 0.9 0.95
VidoreShiftProjectRetrieval 0.84 0.93
VidoreSyntheticDocQAAIRetrieval 0.99 1.00
VidoreSyntheticDocQAEnergyRetrieval 0.96 0.97
VidoreSyntheticDocQAGovernmentReportsRetrieval 0.97 0.98
VidoreSyntheticDocQAHealthcareIndustryRetrieval 0.99 1.00
VidoreTabfquadRetrieval 0.89 0.96
VidoreTatdqaRetrieval 0.68 0.83
Average 0.78 0.84

@KennethEnevoldsen KennethEnevoldsen added the waiting for review of implementation This PR is waiting for an implementation review before merging the results. label Aug 16, 2025
@KennethEnevoldsen
Copy link
Contributor

Results look good - thanks for submitting the results!

@KennethEnevoldsen KennethEnevoldsen merged commit c28a39f into embeddings-benchmark:main Aug 16, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting for review of implementation This PR is waiting for an implementation review before merging the results.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants