Skip to content

Comments

fix: Unsure TaskResults can handle runtime and version being unspecified#1447

Closed
KennethEnevoldsen wants to merge 2 commits intov2.0.0from
update-taskresult-to-allow-None
Closed

fix: Unsure TaskResults can handle runtime and version being unspecified#1447
KennethEnevoldsen wants to merge 2 commits intov2.0.0from
update-taskresult-to-allow-None

Conversation

@KennethEnevoldsen
Copy link
Contributor

see embeddings-benchmark/results#43

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding datasets checklist

Reason for dataset addition: ...

  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding a model checklist

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks.

@KennethEnevoldsen KennethEnevoldsen deleted the update-taskresult-to-allow-None branch November 14, 2024 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant