Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correct padding for SFT input data to account for sequence parallel + TE's fp8 op dimension requirements #8240

Merged
merged 2 commits into from
Jan 25, 2024
Merged

Correct padding for SFT input data to account for sequence parallel + TE's fp8 op dimension requirements #8240

merged 2 commits into from
Jan 25, 2024

Conversation

vysarge
Copy link
Contributor

@vysarge vysarge commented Jan 25, 2024

What does this PR do ?

Alters GPTSFTDataset and GPTSFTPackedDataset to pad to 8*TP when sequence parallel is enabled.

Collection: nlp/language_modeling

Changelog

  • Adds an argument to GPTSFTDataset and GPTSFTPackedDataset which allows setting what value sequence lengths will be padded to a multiple of (previously always 16)
  • Sets this argument from MegatronGPTSFTModel to ensure that sequences will be padded to a multiple of 8*TP when sequence parallel is enabled so that the sequence dimension will remain divisible by 8 after being split (required by TE for the first dimension)
  • Sequences will still be padded to a multiple of 16 when sequence parallel is not used (see PR-7904)

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@vysarge
Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padd…
607bafe 
…ing sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>
@github-actions github-actions bot added the NLP label Jan 25, 2024
@vysarge
Copy link
Contributor Author

vysarge commented Jan 25, 2024

@cuichenx / @ericharper can you review?

ericharper
ericharper approved these changes Jan 25, 2024
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@ericharper
Copy link
Collaborator

jenkins

@ericharper ericharper merged commit 618ff06 into NVIDIA:main Jan 25, 2024
11 checks passed
yaoyu-33 pushed a commit that referenced this pull request Jan 31, 2024
@vysarge @pre-commit-ci @yaoyu-33
Correct padding for SFT input data to account for sequence parallel +…
84d8d6b 
… TE's fp8 op dimension requirements (#8240)

* Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padding sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Valerie Sarge <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
stevehuang52 pushed a commit that referenced this pull request Jan 31, 2024
@vysarge @pre-commit-ci @stevehuang52
Correct padding for SFT input data to account for sequence parallel +…
4a534b8 
… TE's fp8 op dimension requirements (#8240)

* Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padding sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Valerie Sarge <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: stevehuang52 <[email protected]>
ssh-meister pushed a commit to ssh-meister/NeMo that referenced this pull request Feb 15, 2024
@vysarge @pre-commit-ci @sashameister
Correct padding for SFT input data to account for sequence parallel +…
4df1606 
… TE's fp8 op dimension requirements (NVIDIA#8240)

* Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padding sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Valerie Sarge <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Sasha Meister <[email protected]>
@vysarge vysarge deleted the vsarge/sft_sp_fp8_seqlen_pad_fix branch March 12, 2024 22:36
pablo-garay pushed a commit that referenced this pull request Mar 19, 2024
@vysarge @pre-commit-ci @pablo-garay
Correct padding for SFT input data to account for sequence parallel +…
789d7e4 
… TE's fp8 op dimension requirements (#8240)

* Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padding sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Valerie Sarge <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Pablo Garay <[email protected]>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
@vysarge @pre-commit-ci
Correct padding for SFT input data to account for sequence parallel +…
983e178 
… TE's fp8 op dimension requirements (NVIDIA#8240)

* Alter GPTSFTDataset / GPTSFTPackedDataset to account for SP when padding sequences to ensure divisibility by 8/16 for TE with fp8

Signed-off-by: Valerie Sarge <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Valerie Sarge <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericharper ericharper ericharper approved these changes

Assignees
No one assigned
Labels
NLP
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@vysarge @ericharper