Fix issue with Segfault in ASR models #3956
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: smajumdar <[email protected]>
ericharper
pushed a commit
that referenced
this pull request
Apr 8, 2022
* Fix issue with Segfault in ASR models Signed-off-by: smajumdar <[email protected]> * Add docstring Signed-off-by: smajumdar <[email protected]>
titu1994
added a commit
that referenced
this pull request
Apr 9, 2022
* update version Signed-off-by: ericharper <[email protected]> * Stateless timer fix for PTL 1.6 (#3925) * Stateless timer fix for PTL 1.6 Signed-off-by: MaximumEntropy <[email protected]> * Stateless timer PTL test Signed-off-by: MaximumEntropy <[email protected]> * Fix year Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Remove unused imports Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * GPU test Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * clean import Signed-off-by: ericharper <[email protected]> Co-authored-by: ericharper <[email protected]> * fix save_best missing chpt bug, update for setup_tokenizer() changes (#3932) * fix save_best missing chpt bug, update for setup_tokenizer() changes Signed-off-by: ekmb <[email protected]> * style fix Signed-off-by: ekmb <[email protected]> * Fix divide by world size (#3941) Signed-off-by: MaximumEntropy <[email protected]> * remove old doc (#3946) Signed-off-by: ekmb <[email protected]> * Fix issues with librosa deprecations (#3950) Signed-off-by: smajumdar <[email protected]> * Fix issue with Segfault in ASR models (#3956) * Fix issue with Segfault in ASR models Signed-off-by: smajumdar <[email protected]> * Add docstring Signed-off-by: smajumdar <[email protected]> * Fix notebook bugs for branch r1.8.0 (#3948) * load the model from ngc Signed-off-by: Yi Dong <[email protected]> * fix all biomegatron notebook Signed-off-by: Yi Dong <[email protected]> * fix the typos Signed-off-by: Yi Dong <[email protected]> * remove output Signed-off-by: Yi Dong <[email protected]> * fix isort Signed-off-by: Yi Dong <[email protected]> * fix merge error Signed-off-by: Yi Dong <[email protected]> * change ntpath for isort workaround Signed-off-by: Yi Dong <[email protected]> * fix unit test Signed-off-by: Yi Dong <[email protected]> * fix ci Signed-off-by: Yi Dong <[email protected]> * fix ci bert pretraining Signed-off-by: Yi Dong <[email protected]> * make it compatible with main Signed-off-by: Yi Dong <[email protected]> * add the teste for biomegatron ner Signed-off-by: Yi Dong <[email protected]> * fix argument Signed-off-by: Yi Dong <[email protected]> * fix usablity issue Signed-off-by: Yi Dong <[email protected]> * work around Signed-off-by: Yi Dong <[email protected]> Co-authored-by: Yi Dong <[email protected]> Co-authored-by: Eric Harper <[email protected]> * Fix global batch fit loop (#3936) * add lightning module hooks for global batch Signed-off-by: ericharper <[email protected]> * clean scripts Signed-off-by: ericharper <[email protected]> * style Signed-off-by: ericharper <[email protected]> * remove unused import Signed-off-by: ericharper <[email protected]> * DP=1 fix Signed-off-by: MaximumEntropy <[email protected]> * set num dataset workers to 2 Signed-off-by: ericharper <[email protected]> * update validation_loop with GlobalDataFetcher Signed-off-by: ericharper <[email protected]> * add test global data fetcher Signed-off-by: ericharper <[email protected]> * Drop last for test ds Signed-off-by: MaximumEntropy <[email protected]> * Fix test epoch end Signed-off-by: MaximumEntropy <[email protected]> * Style Signed-off-by: MaximumEntropy <[email protected]> * Fix eval Signed-off-by: MaximumEntropy <[email protected]> * Fix reconfigure microbatch in the complete method Signed-off-by: MaximumEntropy <[email protected]> * add comments Signed-off-by: MaximumEntropy <[email protected]> * Set init consumed samples Signed-off-by: MaximumEntropy <[email protected]> * fix shuffle Signed-off-by: MaximumEntropy <[email protected]> * add save_restore_connector arg Signed-off-by: ericharper <[email protected]> * Fix padding for labels and loss mask Signed-off-by: MaximumEntropy <[email protected]> * GLUE/XNLI CI tests Signed-off-by: MaximumEntropy <[email protected]> * limit val batches in hydra fix Signed-off-by: MaximumEntropy <[email protected]> * Restart CI Signed-off-by: MaximumEntropy <[email protected]> * Fix unittest Signed-off-by: MaximumEntropy <[email protected]> Co-authored-by: MaximumEntropy <[email protected]> * Update max_epochs on megatron configs (#3958) * update config Signed-off-by: ericharper <[email protected]> * update config Signed-off-by: ericharper <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * update version Signed-off-by: ericharper <[email protected]> Co-authored-by: Sandeep Subramanian <[email protected]> Co-authored-by: Evelina <[email protected]> Co-authored-by: Somshubra Majumdar <[email protected]> Co-authored-by: Yi Dong <[email protected]> Co-authored-by: Yi Dong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: smajumdar [email protected]
What does this PR do ?
Fixes a segfault in nemo ASR models due to hydra / omegaconf access during DDP.
Collection: [ASR]
Changelog
Before your PR is "Ready for review"
Pre checks:
PR Type: