fix: Update metadata to include active number of parameter to ModelMeta

[ayush1298]

closes #3258

[ayush1298]

Just added csv for now. Will remove memory usage column, add active embedding to ModelMeta of each model and then add column to LB.

[Copilot]

Pull request overview

This pull request adds a CSV file containing model parameter data for models in the MTEB leaderboard. The file documents total parameters, active parameters, and input embedding parameters for over 500 models. This addresses issues #3258 and #3259.

Key Changes

• Addition of a CSV file documenting parameter counts (total, active, and input embedding) for 554 models
• The CSV includes success/error status for each model, tracking whether parameter extraction succeeded or encountered issues

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ayush1298]

@Samoed @KennethEnevoldsen @isaac-chung
1 doubt that I have is, value of n_parameters (total number of parameters) which we have in ModelMeta, is almost equal to(though not exact) same as total_params + embedding_params counted using code.

For model: Alibaba-NLP/gte-Qwen2-7B-instruct

  ModelMeta:  7,613,000,000
  In CSV:  total_params - 7069121024, active_params - 6525621760, embedding_params - 543499264 
so, total_params + embedding_params = 7,612,620,288 which is almost equal to what we have in ModelMeta.

Also in csv these 3 parameters came from below code.

import numpy as np
from transformers import AutoModel

model_name = "google/embeddinggemma-300m"
model = AutoModel.from_pretrained(model_name)

input_params = np.prod(model.get_input_embeddings().weight.shape)
total_params = np.sum([np.prod(x.shape) for x in list(model.parameters())])
active_params = total_params - input_params

So, what exactly, I should add in ModelMeta? Should I add 2 new field in ModelMeta with name active_parameter, and embedding_parameter, and should we show both of these on LB, and remove memory_usage and number of parameters?

[Samoed]

Let's modify leaderboard in separate PR

1 doubt that I have is, value of n_parameters (total number of parameters) which we have in ModelMeta, is almost equal to(though not exact) same as total_params + embedding_params counted using code.

Yes, this is expected that total parameters = active parameters + embedding paramters

Should I add 2 new field in ModelMeta with name active_parameter, and embedding_parameter

Probably yes

should we show both of these on LB, and remove memory_usage and number of parameters?

Let's discuss this in issue

[KennethEnevoldsen]

Converting this to a draft given the current state, but great to see it!

• Should we have a script to compute active parameters (e.g. the automatic metadata). Should we consider how we handle MoE? (this is where active != total - embedding)
• Leaderboard: Let us deal with that in a seperate PR
• embedding parameters: If we define active parameters as total - embedding (might be a good enough assumption) I would probably just add embedding parameters and let the other one be a property. We could potentially allow an overwrite using e.g. self._active_parameters

[ayush1298]

• Should we have a script to compute active parameters (e.g. the automatic metadata). Should we consider how we handle MoE? (this is where active != total - embedding)

Yes, I will be adding the logic to calculate automatically in from_hub method. I think MOE are case, where I was not able to calculate it. Hence I think atleast in ModelMeta we can keep n_parameters and add both active and embedding. Also, as I have already calculated these parameters for around 315 models, and around 60 models are propritory out of 498 models we have currently.

• embedding parameters: If we define active parameters as total - embedding (might be a good enough assumption) I would probably just add embedding parameters and let the other one be a property. We could potentially allow an overwrite using e.g. self._active_parameters

For now, we can use csv. But, I think its better to keep both in ModelMeta as then we could also handle it for MOE case, where these property will not be useful.

[ayush1298]

I have updated files with these meta and also added it to .from_hub() method. I have also added script that I have used to update ModelMeta. Will remove these script at end.

[ayush1298]

Previously when I was doing make lint, it was fixing errors itself (before adding uv everywhere), but now somehow it was just giving error, but not fixing them. Why this was happening?

[Samoed]

Strange. I don't have this behavior

[KennethEnevoldsen]

Hmm do we want to use the csv? I think we should rather use the ModelMeta object - if not then I think we should use the json format similar to what we do with descriptive statistics

[Samoed]

I think csv just for demonstration

[ayush1298]

Hmm do we want to use the csv? I think we should rather use the ModelMeta object - if not then I think we should use the json format similar to what we do with descriptive statistics

Its just for demonstration of all results I have calculated, and I had filled ModelMeta of models using that csv.
For future if someone wants to calculate or add it for models where I am not able to calculate, then can just do it in below way:

model_meta = mteb.get_model_meta("model_name")
active_parameters, embedding_parameters  = model_meta.extract_parameter_breakdown_from_hub

[KennethEnevoldsen]

I would convert active parameters to a property and just add embedding parameters.

[ayush1298]

Converted active_parameters to property. I will check for which models n_parameter != embedding + active based on results in csv, and only for them will keep _n_active_parameters field in ModelMeta and will remove these field for rest of all models.

[Samoed]

I don't think we can do anything with it

[KennethEnevoldsen]

I added a few suggestions - I think that should finalize this PR (thanks for taking the time on it)

[ayush1298]

I have deleted csv and created issue to fill n_embedding_parameters (#3947 ) and to calculate active_parameters for MOE models (#3948)

[KennethEnevoldsen]

@ayush1298 will to resolve the merge - I think this is good to go. @Samoed I marked it as fix since it is only an update on the metadata, but would be fine with doing a feat as well.