Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix ptl_bugs in slu_models.py #7689

Merged
merged 5 commits into from
Oct 13, 2023

Conversation

jzi040941
Copy link
Contributor

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

This PR added fixes for PTL2.0 related SLU bugs in r1.21.0: Val metrics logging, None dataloader issue

Collection:
ASR

Changelog

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

@stevehuang52 @titu1994

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Signed-off-by: Seonghun Noh <[email protected]>
@github-actions github-actions bot added the ASR label Oct 11, 2023
@stevehuang52 stevehuang52 self-assigned this Oct 12, 2023
Copy link
Collaborator

@stevehuang52 stevehuang52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix~!

Could you please also change the trainer.strategy=ddp to trainer.strategy=ddp_find_unused_parameters_true in the yaml file? It was giving the following error when I ran with ddp:

RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step. If this is intentional, you must enable the detection of unused parameters in DDP, either by setting the string value `strategy='ddp_find_unused_parameters_true'` or by setting the flag in the strategy with `strategy=DDPStrategy(find_unused_parameters=True)`.

@jzi040941 did you experience the same issue?

I'll approve once this is resolved.

@jzi040941
Copy link
Contributor Author

@stevehuang52 Yes, I had same issue with ddp strategy,
it works well after change it to ddp_find_unused_parameters_true

I updated the yaml file, thanks~

Copy link
Collaborator

@stevehuang52 stevehuang52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks~!

@stevehuang52 stevehuang52 merged commit 9c955c4 into NVIDIA:r1.21.0 Oct 13, 2023
9 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 13, 2023
* fix ptl_bugs in slu_models.py

Signed-off-by: Seonghun Noh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change strategy to ddp_find_unused_parameters_true in  slu example yaml

Signed-off-by: Seonghun <[email protected]>

---------

Signed-off-by: Seonghun Noh <[email protected]>
Signed-off-by: Seonghun <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <[email protected]>
titu1994 pushed a commit that referenced this pull request Oct 13, 2023
* fix ptl_bugs in slu_models.py



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change strategy to ddp_find_unused_parameters_true in  slu example yaml



---------

Signed-off-by: Seonghun Noh <[email protected]>
Signed-off-by: Seonghun <[email protected]>
Co-authored-by: Seonghun Noh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <[email protected]>
yaoyu-33 pushed a commit that referenced this pull request Oct 16, 2023
* fix ptl_bugs in slu_models.py



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change strategy to ddp_find_unused_parameters_true in  slu example yaml



---------

Signed-off-by: Seonghun Noh <[email protected]>
Signed-off-by: Seonghun <[email protected]>
Co-authored-by: Seonghun Noh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <[email protected]>
pzelasko pushed a commit to pzelasko/NeMo that referenced this pull request Jan 3, 2024
* fix ptl_bugs in slu_models.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change strategy to ddp_find_unused_parameters_true in  slu example yaml

---------

Signed-off-by: Seonghun Noh <[email protected]>
Signed-off-by: Seonghun <[email protected]>
Co-authored-by: Seonghun Noh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
* fix ptl_bugs in slu_models.py



* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change strategy to ddp_find_unused_parameters_true in  slu example yaml



---------

Signed-off-by: Seonghun Noh <[email protected]>
Signed-off-by: Seonghun <[email protected]>
Co-authored-by: Seonghun Noh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: He Huang (Steve) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants