Conversation
|
Have you already discussed with @KennethEnevoldsen or @x-tabdeveloping ? I'd prefer not to have such back-and-forth: to add then remove then add back things. #1729 |
|
Opened issue for this #1935. Previously it was removed, because it was not filled and should be autocalculated based on the number of parameters of the model |
x-tabdeveloping
left a comment
x-tabdeveloping
left a comment
There was a problem hiding this comment.
Left a couple of minor comments. Thanks for adding this
mteb/model_meta.py
Outdated
| if self.n_parameters is None: | ||
| return None | ||
| # Model memory in bytes. For FP32 each parameter is 4 bytes. | ||
| model_memory_bytes = self.num_params * 4 |
There was a problem hiding this comment.
Is this a good assumption to make? Do all models have FP32 parameters?
There was a problem hiding this comment.
Large models (>1B) params usally loaded with fp16/bp16, but I don't know how to handle this automatically
There was a problem hiding this comment.
I suppose you could get this information using huggingface_hub.hf_api.get_safetensors_metadata
There was a problem hiding this comment.
And then do a cached_property so that it doesn't have to be fetched every time
There was a problem hiding this comment.
I think integrating this could slow down leaderboard building. Maybe we could manually set memory usage instead or add information about the number of parameters for each model weight?
There was a problem hiding this comment.
I agree. We could fetch all of these in a script and manually keep count of them in ModelMeta perhaps?
# Conflicts: # mteb/models/bge_models.py # mteb/models/promptriever_models.py
# Conflicts: # mteb/models/ru_sentence_models.py
|
@x-tabdeveloping Can this PR be merged? |
KennethEnevoldsen
left a comment
KennethEnevoldsen
left a comment
There was a problem hiding this comment.
Looks good on my end - great addition. Do we want to create a column for it on the leaderboard (if so let us make an issue on that)
|
@KennethEnevoldsen I've already created issue #1935 |
# Conflicts: # mteb/models/gritlm_models.py
4 participants
To make fully compatible old leaderboard with new, we can add model memory usage. Add property to
ModelMeta.Code Quality
make lintto maintain consistent style.Documentation
Testing
make test-with-coverage.make testormake test-with-coverageto ensure no existing functionality is broken.