Skip to content

Add unfused clap model for zero-shot#2269

Merged
RahulSChand merged 2 commits intoembeddings-benchmark:maebfrom
anime-sh:zero_shot
Mar 6, 2025
Merged

Add unfused clap model for zero-shot#2269
RahulSChand merged 2 commits intoembeddings-benchmark:maebfrom
anime-sh:zero_shot

Conversation

@RahulSChand
Copy link

This is a small addition to the #2133 PR. Adds unfused clap model. Test results in the next comment

Code Quality

  • [ ✅] Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • [ ✅] Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • [ ❌] New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • [ ❌] Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding datasets checklist

Reason for dataset addition: ...

  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding a model checklist

  • [ ✅] I have filled out the ModelMeta object to the extent possible
  • [ ✅] I have ensured that my model can be loaded using
    • [ ✅] mteb.get_model(model_name, revision) and
    • [ ✅] mteb.get_model_meta(model_name, revision)
  • [ ✅] I have tested the implementation works on a representative set of tasks.

@RahulSChand RahulSChand added the maeb Audio extension label Mar 6, 2025
@RahulSChand RahulSChand self-assigned this Mar 6, 2025
@RahulSChand
Copy link
Author

RahulSChand commented Mar 6, 2025

Tested with on ESC50

import mteb
model_name = "laion/clap-htsat-unfused"
model = mteb.get_model_meta(model_name=model_name)

tasks = mteb.get_tasks(tasks=["ESC50_Zeroshot"])

evaluation = mteb.MTEB(tasks=tasks)

results = evaluation.run(model)
print(results[0].scores)

Screenshot from 2025-03-11 12-29-10

@RahulSChand RahulSChand requested a review from Samoed March 6, 2025 11:25
@RahulSChand RahulSChand merged commit 6d9eca3 into embeddings-benchmark:maeb Mar 6, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maeb Audio extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants