Skip to content

Conversation

@w601sxs
Copy link
Contributor

@w601sxs w601sxs commented Mar 12, 2025

Checklist

  • [ X ] Run tests locally to make sure nothing is broken using make test.
  • [ X ] Run the results files checker make pre-push.

Adding a model checklist

  • [ X ] I have added model implementation to mteb/models/ directory. Instruction to add a model can be found here in the following PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need a revision as well as an updated model meta

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this just the output of ModelMeta.dict() ?

{'name': 'b1ade-embed',
 'revision': '3bdac13927fdc888b903db93b2ffdbd90b295a69',
 'release_date': '2025-03-10',
 'languages': ['eng-Latn'],
 'loader': None,
 'n_parameters': 335000000,
 'memory_usage_mb': 1278.0,
 'max_tokens': 4096.0,
 'embed_dim': 1024,
 'license': 'mit',
 'open_weights': True,
 'public_training_code': None,
 'public_training_data': None,
 'framework': ['Sentence Transformers', 'PyTorch'],
 'reference': 'https://huggingface.co/w601sxs/b1ade-embed',
 'similarity_fn_name': 'cosine',
 'use_instructions': False,
 'training_datasets': {'MSMARCO': []},
 'adapted_from': None,
 'superseded_by': None,
 'is_cross_encoder': None,
 'modalities': ['text']}

would that do? In that case I will use this revision id

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, added based on another folder

@w601sxs
Copy link
Contributor Author

w601sxs commented Mar 12, 2025

I see AssertionError: assert 'no_revision_available' not in ['no_revision_available', 'external'] but other model folders that are merged do have both ?? LMK what I need to do

@Samoed
Copy link
Member

Samoed commented Mar 12, 2025

You should put your results in the folder with revision of your model.

@w601sxs
Copy link
Contributor Author

w601sxs commented Mar 12, 2025

^ had to remove .ipynb_checkpoint, but looks like checks passed now. Sorry for the sequence of commits

@KennethEnevoldsen KennethEnevoldsen merged commit c1d06e9 into embeddings-benchmark:main Mar 19, 2025
2 checks passed
@w601sxs
Copy link
Contributor Author

w601sxs commented Mar 20, 2025

Does the leaderboard refresh every week/ every day? Wondering why the model does not show up on the leaderboard @KennethEnevoldsen 🔢

@Samoed
Copy link
Member

Samoed commented Mar 20, 2025

I think it was added in embeddings-benchmark/mteb#2386 with incorrect name (witout user/org prefix). I'll fix that

@Samoed
Copy link
Member

Samoed commented Mar 20, 2025

Now it's showing correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants