Skip to content

Added Wav2Vec model, voice clustering task, VoxCeleb dataset subset (…#1

Merged
sufen-f merged 2 commits intosufen-f:maebfrom
embeddings-benchmark:maeb
Feb 28, 2025
Merged

Added Wav2Vec model, voice clustering task, VoxCeleb dataset subset (…#1
sufen-f merged 2 commits intosufen-f:maebfrom
embeddings-benchmark:maeb

Conversation

@sufen-f
Copy link
Owner

@sufen-f sufen-f commented Feb 28, 2025

embeddings-benchmark#2175)

  • Added wav2vec model wrapper

  • Added four w2v variants

  • Update wav2vec_models.py

  • Removed run.py test script

  • Added subTask with small sample of dataset for testing

  • Removed test portion of VoiceGender.py task

  • add commit hash and bibtex

  • make lint

  • update models

  • fix circular import

  • make VoiceGender discoverable in get_tasks

  • add a2a as category for clustering

  • specify latest commit hash

  • revert linting changes

  • Based on feedback for model: updated w2v2 revisions and added torchaudio to .toml file

  • Added Bibtex for dataset, set data to be test instead of training, shortened task_subtype

  • Changed task from Voice Gender Clustering to Gender Clustering.

  • Fixed mock audio clustering tests

  • Added dataset metadata

  • Linted

  • Passed revision into the w2v2 loader

  • passed lint check

  • Linted

  • Update VoiceGender.py


Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding datasets checklist

Reason for dataset addition: ...

  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding a model checklist

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks.

sufen-f and others added 2 commits February 28, 2025 10:49
…2175)

* Added wav2vec model wrapper

* Added four w2v variants

* Update wav2vec_models.py

* Removed run.py test script

* Added subTask with small sample of dataset for testing

* Removed test portion of VoiceGender.py task

* add commit hash and bibtex

* make lint

* update models

* fix circular import

* make VoiceGender discoverable in get_tasks

* add a2a as category for clustering

* specify latest commit hash

* revert linting changes

* Based on feedback for model: updated w2v2 revisions and added torchaudio to .toml file

* Added Bibtex for dataset, set data to be test instead of training, shortened task_subtype

* Changed task from Voice Gender Clustering to Gender Clustering.

* Fixed mock audio clustering tests

* Added dataset metadata

* Linted

* Passed revision into the w2v2 loader

* passed lint check

* Linted

* Update VoiceGender.py

---------

Co-authored-by: Ali Sartaz Khan <alisartazkhan@gmail.com>
Co-authored-by: Ali Sartaz Khan <71156712+alisartazkhan@users.noreply.github.com>
Co-authored-by: mn <mn@Ms-MacBook-Pro.local>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
@sufen-f sufen-f merged commit 00edb54 into sufen-f:maeb Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant