Add FlaxWhisperForAudioClassification model by raghavanone · Pull Request #21894 · huggingface/transformers

raghavanone · 2023-03-02T07:53:14Z

What does this PR do?

Please review and let me know changes @sanchit-gandhi

HuggingFaceDocBuilderDev · 2023-03-02T09:15:42Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

sanchit-gandhi

Modelling code looks good @raghavanone! Nice one on getting this working so quickly 🙌 Do you want to have a go at adding the encoder-only tests? See the PyTorch WhisperForAudioClassficiation PR for details, think you can also add these quite quickly :)

raghavanone · 2023-03-03T12:24:45Z

Modelling code looks good @raghavanone! Nice one on getting this working so quickly 🙌 Do you want to have a go at adding the encoder-only tests? See the PyTorch WhisperForAudioClassficiation PR for details, think you can also add these quite quickly :)

I have added the Encoder tests, But some test are failing, The FlaxWhisperForAudioClassification class extends FlaxWhisperPreTrainedModel . Due to this inheritance, the call method expects decoder related params.

Should the FlaxWhisperForAudioClassification not extend FlaxWhisperPreTrainedModel instead create a new pretrainedclass ?

sanchit-gandhi · 2023-03-07T15:22:37Z

Hey @raghavanone! The PyTorch model has just been merged (#21754), so you can rebase onto main to get the required config changes:

git fetch upstream
git rebase upstream/main

This will fix the failing Flax tests we're getting here: https://app.circleci.com/pipelines/github/huggingface/transformers/58972/workflows/2388bd70-553e-412f-9ee7-0599cace5639/jobs/719829

The only thing to make sure is that the first time you push after rebasing, you force push to origin:

git add .
git commit -m "Some new changes after rebase"
git push -f origin fix_issue_21779

You only have to force push once, the next time you can just regular push:

git add .
git commit -m "Some more changes"
git push -u origin fix_issue_21779

raghavanone · 2023-03-16T09:54:34Z

@sanchit-gandhi There are 2 test failing here, I am unable to get the same failure locally in my machine. Any pointers on how to replicate failing test and fix it ?

sanchit-gandhi · 2023-03-23T16:00:34Z

+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            rngs=rngs,
+            # method=_encoder_forward,


Can remove this commented line too

sanchit-gandhi

Nice work @raghavanone! Mainly just some clean-up before we can get this merged!

sanchit-gandhi

Nice one @raghavanone! Mainly just some code clean-up, then we can get this merged!

sanchit-gandhi · 2023-04-04T16:35:06Z

Hey @raghavanone! Would you mind going through the previous review comments and marking them as resolved where you've addressed them? I'll then get you a final review asap! Thanks!

* Add BridgeTower for ITC * Fix review feedback * Rename BridgeTowerForITC, cleanup * Fix style and quality * implement tests --------- Co-authored-by: Tiep Le <97980157+tileintel@users.noreply.github.com> Co-authored-by: Tiep Le <tiep.le@intel.com>

…line (huggingface#22031) add tokenize_kwargs doc in the FeatureExtractionPipeline

…on_seq2seq.py (huggingface#21942) * Add specaugment to run_speech_recognition_seq2seq.py * Remove useless argument: text_column * Fix quality * Update return_attention_mask condition * Update specaugment arguments only for whisper models * Remove SpecAugment arguments from ModelArguments, only leave default values for simplicity * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Update apply_spec_augment only for whisper models * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Rename return_attention_mask to forward_attention_mask to avoid confusion with wav2vec2 models --------- Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* fixing * Update modeling_whisper.py * Update modeling_whisper.py * Update src/transformers/models/whisper/modeling_whisper.py --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

…P-like models (huggingface#22035) * Avoid text_config_dict and vision_config_dict being saved * for other CLIP-like models --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* slow me --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

…uggingface#22034) fix slow tokenizers with passing offset_mapping

* Fix typos and add code examples, resources

* [21737][T5]: Fix gradient checkpoint bug * [21737][T5]: Fix gradient checkpoint bug * [21737][T5]: Fix gradient checkpoint bug * Update src/transformers/models/mt5/modeling_mt5.py * Update src/transformers/models/t5/modeling_t5.py --------- Co-authored-by: njindal <njindal@adobe.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

…x it (huggingface#22045) In ZSH, not using ' ' around pip install fails Running ``` pip install transformers[torch] ``` in the default ZSH terminal will fail with the error `zsh: no matches found: transformers[torch]` The solution is to wrap the installation path in ' ' like ``` pip install 'transformers[torch]' ``` Relevant StackOverflow: https://stackoverflow.com/questions/30539798/zsh-no-matches-found-requestssecurity

…ace#22051) * Remove set_access_token usage + fail tests if FutureWarning * do not fail on FutureWarning in CI --------- Co-authored-by: testbot <lucainp@hf.co>

…ce#22054) * show hfh warnings --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

…ce#22040) * return analysis for hyperparameter_search with ray backend * Revert "return analysis for hyperparameter_search with ray backend" This reverts commit cd51790. * add run_summary attribute to BestRun and return analysis for ray backend * fix typo * add doc for run_summary for ray backend

* Add an argument to pt-to-tf to allow overriding the model class * make fixup * Minor fix to error message * Remove unused extra conversion from the script

rm $ symbol from code block Removed the $ symbol from the code block to make copy-pasting easier.

) * [deepspeed] offload + non-cpuadam optimizer exception * flip * revert min version

…face#22033) * Edit the docstring of `image_processing_donut` to match code * improve style * more style improvement after installing quality

…e#21695) LayoutLMv3TokenizerFast produces empty 'Ġ' token with `offset_mapping = (0, 0)`. Next token is wrongly assumed to also be beginning of word and isn't correctly assigned `pad_token_label`. Modify test with text that produce 'Ġ' token. Remove copy check from LayoutLMv2TokenizerFast for `_batch_encode_plus`. solves issue: huggingface#19978

…e GPUs using `accelerate` (huggingface#22532) * add `is_model_parallel` arg on Trainer * add warning * adapt from suggestions * revert t5 changes * remove commas * adapt from suggestions

…#22535) * enable PP for T5 * make fixup * fix failing tests

* [setup] drop deprecated `distutils` usage * drop deprecated `distutils.util.strtobool` usage * fix import order * reformat docstring by `doc-builder`

* [setup] migrate setup script to `pyproject.toml` * [setup] cleanup configurations * remove unused imports

* Fix OPTForQuestionAnswering doc string for more adequate model answer decoding * black style fix * doc-builder style

…length of past_key_values when generating as a decoder (huggingface#22416) * fix RoFormerEncoder postion embedding when generate as decoder * make fixup * add test case for check generate with past key values * remove duplicating code

* fix the prefix tokens * update fast and test values * add legacy behaviour Co-authored-by: sgugger <sylvain.gugger@gmail.com> * update disclaimer, linkissue PR and behaviral changes * Apply suggestions from code review Co-authored-by: Lysandre Debut <hi@lysand.re> * styling * make a quote * quote this time --------- Co-authored-by: sgugger <sylvain.gugger@gmail.com> Co-authored-by: Lysandre Debut <hi@lysand.re>

…e#22498) * implemented safetensors save/load * remove duplicated file * added tests * more tests * style fix * fix tf tests * change to list comprehension Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * review fixes + safe load for sharded checkpoint * style fix * remove rogue import * remove partial to avoid undefined exception * use naming alias instead of safetensors.torch * fix safe sharding in tests * grammar Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * update docs Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * update docs Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * minor corrections * style --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

…gingface#22537)

Update modeling_utils.py

…22558) Add id2label and label2id to config in run_xnil

* Soft error whisper. * Fix format. --------- Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-94.taildb5d.ts.net>

* Initial commit * more stash commit * Yet another stash commit * yet more stash commit * Mostly working except for docs / repo consistency * Stop importing model list from torch file * Add TF BLIP models to docs * Add auto classes * Move get_text_features and get_image_features * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip_text.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update tests/models/blip/test_modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update tests/models/blip/test_modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update tests/models/blip/test_modeling_tf_blip_text.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip_text.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Use channels_last convolutions in TF (better performance + compatibility) * Remove _shape function * Move multi-line statement to one line in PT + TF * Specify tf.keras.layers instead of importing from it * Remove test_gradient_checkpointing and empty test_training methods * move some multi-line statements to one line * Update docstring for generate * Remove pruned heads set * Remove self.seq_len_dim * Fixed issues with loss computation, should resolve some tests. Also ensured that the PT version follows the config for output_attentions and output_hidden_states * ensure original model follows config in more cases * Skip the same cross-attention tests in the PT tests - didn't realize we did it twice! * Add training args throughout the models and layers * make fixup * Fix docstring for inputs_embeds * Add docstring for is_decoder * Add docstrings to text models * Remove redundant computation * Add unpack_inputs / keras_serializable * Add modeling_tf_blip to doctests * Add config classes for keras serialization * Changes to allow model porting with pt-to-tf * Quick fix to decoder head and test tweaks * Revert an issue with masking the embeddings outputs * Allow missing keys in some equivalence tests (for unused layers) * Add tf-pt equivalence tests back in * Update src/transformers/models/blip/modeling_tf_blip.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip_text.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/blip/modeling_tf_blip_text.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * make fixup * Refactor invert_attention_mask out into tf_utils * Re-enable cross-tests on the PT side too --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

…_indices (huggingface#22557) * corrected/clarified the code comment of find_pruneable_heads_and_indices * have run make style

* initial commit * review changes * post model PR merge * updating doc

* Fix inverted conditional in TF common test! * Make the same change in the PT tests file * Make sure hidden states for GPT2 have the same output shape in PT/TF * Minor fix to PT implementation of token classification loss * Skip loss equivalence test for TFHubert because it keeps overflowing to inf * Compute LM loss for TF the (weird) way it's computed in PT * Skip loss equivalence test for Wav2Vec2 for the same reason as Hubert * Fix - don't try to access the hidden states property when output is a tuple

sanchit-gandhi · 2023-04-18T16:45:51Z

Hey @raghavanone - I think the commit history has been corrupted for this PR? Gentle reminder that one must force push after rebasing: #21894 (comment) Think this is probably the culprit for the 250 extra commits!

In this instance, it's probably best to close this PR in favour of a new one that only contains the new changes you with to merge. Sorry about that!

sanchit-gandhi · 2023-05-04T15:44:58Z

Closing in favour of #22883

sanchit-gandhi reviewed Mar 2, 2023

View reviewed changes

raghavanone force-pushed the fix_issue_21779 branch from 2561cce to 7cb70bd Compare March 7, 2023 23:58

sanchit-gandhi reviewed Mar 9, 2023

View reviewed changes

Comment thread src/transformers/models/whisper/modeling_flax_whisper.py Outdated

sanchit-gandhi reviewed Mar 23, 2023

View reviewed changes

abhiwand and others added 19 commits April 5, 2023 12:47

Fix test for torchneuroncore in Trainer (huggingface#22028)

fe03e51

Add tokenize_kwargs parameter definition in the FeatureExtractionPipe…

781edc5

…line (huggingface#22031) add tokenize_kwargs doc in the FeatureExtractionPipeline

fixes the gradient checkpointing of whisper (huggingface#22019)

9a10d4b

* fixing * Update modeling_whisper.py * Update modeling_whisper.py * Update src/transformers/models/whisper/modeling_whisper.py --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

Avoid text_config_dict and vision_config_dict being saved for CLI…

65f56ec

…P-like models (huggingface#22035) * Avoid text_config_dict and vision_config_dict being saved * for other CLIP-like models --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Mark all BridgeTower tests slow for now (huggingface#22039)

3f29db2

* slow me --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Bug fix: token classification pipeline while passing offset_mapping (h…

2a5f185

…uggingface#22034) fix slow tokenizers with passing offset_mapping

Update ALIGN docs (huggingface#22025)

f89b95c

* Fix typos and add code examples, resources

Can't install tf2 on M1 Chip by default (huggingface#22046)

c2420fd

Remove set_access_token usage + fail tests if FutureWarning (huggingf…

da98339

…ace#22051) * Remove set_access_token usage + fail tests if FutureWarning * do not fail on FutureWarning in CI --------- Co-authored-by: testbot <lucainp@hf.co>

Show the number of huggingface_hub warnings in CI report (huggingfa…

6fd5f2f

…ce#22054) * show hfh warnings --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

pt-to-tf model architecture override (huggingface#22055)

a7dacfb

* Add an argument to pt-to-tf to allow overriding the model class * make fixup * Minor fix to error message * Remove unused extra conversion from the script

rm $ symbol from code block from contributing.md (huggingface#22057)

c8437ea

rm $ symbol from code block Removed the $ symbol from the code block to make copy-pasting easier.

[deepspeed] offload + non-cpuadam optimizer exception (huggingface#22043

09e9344

) * [deepspeed] offload + non-cpuadam optimizer exception * flip * revert min version

Edit the docstring of image_processing_donut to match code (hugging…

e39722e

…face#22033) * Edit the docstring of `image_processing_donut` to match code * improve style * more style improvement after installing quality

python273 and others added 26 commits April 5, 2023 12:47

llama docs: fix conversion script url (huggingface#22514)

af961bf

[BLIP] fix cross attentions for BlipTextEncoder (huggingface#22515)

fe24b03

[Trainer] Force is_model_parallel when model is loaded in multipl…

0d419e1

…e GPUs using `accelerate` (huggingface#22532) * add `is_model_parallel` arg on Trainer * add warning * adapt from suggestions * revert t5 changes * remove commas * adapt from suggestions

[T5] Enable naive Pipeline Parallelism training for T5 (huggingface…

1376d60

…#22535) * enable PP for T5 * make fixup * fix failing tests

Fix missing metrics with multiple eval datasets (huggingface#22536)

7adc163

[setup] drop deprecated distutils usage (huggingface#22531)

2cccab4

* [setup] drop deprecated `distutils` usage * drop deprecated `distutils.util.strtobool` usage * fix import order * reformat docstring by `doc-builder`

Generate: Enable easier TextStreamer customization (huggingface#22516)

a02907d

[setup] migrate setup script to pyproject.toml (huggingface#22539)

ecac59c

* [setup] migrate setup script to `pyproject.toml` * [setup] cleanup configurations * remove unused imports

Skip failing test

d5372fd

Update test_image_processing_pix2struct.py (huggingface#22543)

14f22d1

Fix OPTForQuestionAnswering doc string (huggingface#22481)

889607f

* Fix OPTForQuestionAnswering doc string for more adequate model answer decoding * black style fix * doc-builder style

Generate: Add text streamer decoding options (huggingface#22544)

995ad7b

Remove hack for dynamic modules and use Python functions instead (hug…

19ce467

…gingface#22537)

[bnb] Fix typo (huggingface#22556)

3a1e4b7

Update modeling_utils.py

Add id2label and label2id to model's config in run_xnil (huggingface#…

763a78e

…22558) Add id2label and label2id to config in run_xnil

Soft error whisper. (huggingface#22475)

dad2f7f

* Soft error whisper. * Fix format. --------- Co-authored-by: Ubuntu <ubuntu@ip-172-31-34-94.taildb5d.ts.net>

corrected the code comment for the output of find_pruneable_heads_and…

271bc94

…_indices (huggingface#22557) * corrected/clarified the code comment of find_pruneable_heads_and_indices * have run make style

Flax Regnet (huggingface#21867)

10dc0e1

* initial commit * review changes * post model PR merge * updating doc

fix _no_split_modules for Whisper model (huggingface#22486)

f4070f7

Skip failing test

05efd03

sanchit-gandhi closed this May 4, 2023

sanchit-gandhi mentioned this pull request May 4, 2023

Add FlaxWhisperForAudioClassification model #22883

Merged

Conversation

raghavanone commented Mar 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Mar 2, 2023

Uh oh!

sanchit-gandhi left a comment

Choose a reason for hiding this comment

Uh oh!

raghavanone commented Mar 3, 2023

Uh oh!

sanchit-gandhi commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

raghavanone commented Mar 16, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sanchit-gandhi Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sanchit-gandhi left a comment

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi left a comment

Choose a reason for hiding this comment

Uh oh!

sanchit-gandhi commented Apr 4, 2023

Uh oh!

sanchit-gandhi commented Apr 18, 2023

Uh oh!

sanchit-gandhi commented May 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

raghavanone commented Mar 2, 2023 •

edited

Loading

sanchit-gandhi commented Mar 7, 2023 •

edited

Loading