Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ALiBi Positional Embeddings #5467

Merged
merged 48 commits into from
Jan 4, 2023
Merged

ALiBi Positional Embeddings #5467

merged 48 commits into from
Jan 4, 2023

Conversation

michalivne
Copy link
Collaborator

@michalivne michalivne commented Nov 21, 2022 •
edited
Loading

Adding support of ALiBi positional embeddings to Megatron
https://arxiv.org/abs/2108.12409

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@lgtm-com
Copy link

lgtm-com bot commented Nov 21, 2022

This pull request introduces 1 alert when merging bfa6217 into 8f69744 - view on LGTM.com

new alerts:

  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@SeanNaren SeanNaren added feature request/PR for a new feature NLP labels Nov 22, 2022
@lgtm-com
Copy link

lgtm-com bot commented Nov 22, 2022

This pull request introduces 3 alerts when merging de37c7b into f8f594e - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@okuchaiev okuchaiev requested a review from arendu December 1, 2022 19:22
@lgtm-com
Copy link

lgtm-com bot commented Dec 1, 2022

This pull request introduces 3 alerts when merging 9756a6e into 5c1d59e - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@lgtm-com
Copy link

lgtm-com bot commented Dec 5, 2022

This pull request introduces 3 alerts when merging 2ed62c2 into fa46393 - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@lgtm-com
Copy link

lgtm-com bot commented Dec 5, 2022

This pull request introduces 3 alerts when merging ce5b369 into 626a4b4 - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@lgtm-com
Copy link

lgtm-com bot commented Dec 6, 2022

This pull request introduces 3 alerts when merging 06d2dd6 into 786a850 - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

@lgtm-com
Copy link

lgtm-com bot commented Dec 6, 2022

This pull request introduces 3 alerts when merging b62b234 into 2a61014 - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

michalivne and others added 4 commits December 6, 2022 15:03
@michalivne
1. Debugging.
45dc27b 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
1. Debugging.
c9224bf 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
Merge branch 'megatron-positiona-encoding-alibi' of github.com:NVIDIA…
63ab313 
…/NeMo into megatron-positiona-encoding-alibi

Signed-off-by: Micha Livne <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c909f50 
for more information, see https://pre-commit.ci
@github-actions github-actions bot removed the NLP label Dec 20, 2022
@github-actions github-actions bot added the NLP label Dec 20, 2022
@github-actions github-actions bot added the CI label Dec 21, 2022
michalivne and others added 12 commits December 20, 2022 16:03
@michalivne
1. Added missing file.
79ca1bd 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
88dbbfd
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
62283bd 
for more information, see https://pre-commit.ci
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
399a932
@michalivne
1. Debugging.
1b04afe 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
Merge branch 'megatron-positiona-encoding-alibi' of github.com:NVIDIA…
a0f817d 
…/NeMo into megatron-positiona-encoding-alibi
@michalivne
1. Debugging.
fc594da 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
f217653
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
80676ed
@michalivne
1. Debugging.
f7ce63d 
Signed-off-by: Micha Livne <[email protected]>
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
68a8e3f
@michalivne
Merge branch 'main' into megatron-positiona-encoding-alibi
c4fb943
ericharper
ericharper approved these changes Dec 28, 2022
View reviewed changes
Copy link
Collaborator

@ericharper ericharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@michalivne michalivne merged commit eee715f into main Jan 4, 2023
andrusenkoau pushed a commit to andrusenkoau/NeMo that referenced this pull request Jan 5, 2023
@michalivne @pre-commit-ci
@okuchaiev @ericharper @andrusenkoau
ALiBi Positional Embeddings (NVIDIA#5467)
32b34ef 
* 1. Working on alibi positional embeddings.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added encoder and decoder alibi classes.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Simplified code.
2. Added bidirectional support.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added support in config to alibi.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added Jenkins tests.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added missing file.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

Signed-off-by: Micha Livne <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: andrusenkoau <[email protected]>
@michalivne michalivne deleted the megatron-positiona-encoding-alibi branch January 5, 2023 23:28
erastorgueva-nv pushed a commit that referenced this pull request Jan 12, 2023
@michalivne @pre-commit-ci
@okuchaiev @ericharper @erastorgueva-nv
ALiBi Positional Embeddings (#5467)
a91e7e3 
* 1. Working on alibi positional embeddings.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added encoder and decoder alibi classes.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Simplified code.
2. Added bidirectional support.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added support in config to alibi.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added Jenkins tests.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added missing file.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

Signed-off-by: Micha Livne <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Signed-off-by: Elena Rastorgueva <[email protected]>
titu1994 pushed a commit to titu1994/NeMo that referenced this pull request Mar 24, 2023
@michalivne @pre-commit-ci
@okuchaiev @ericharper @titu1994
ALiBi Positional Embeddings (NVIDIA#5467)
b269753 
* 1. Working on alibi positional embeddings.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added encoder and decoder alibi classes.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Simplified code.
2. Added bidirectional support.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added support in config to alibi.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Added Jenkins tests.

Signed-off-by: Micha Livne <[email protected]>

* 1. Added missing file.

Signed-off-by: Micha Livne <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

* 1. Debugging.

Signed-off-by: Micha Livne <[email protected]>

Signed-off-by: Micha Livne <[email protected]>
Co-authored-by: Micha Livne <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Oleksii Kuchaiev <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericharper ericharper ericharper approved these changes

@arendu arendu Awaiting requested review from arendu

@MaximumEntropy MaximumEntropy Awaiting requested review from MaximumEntropy

Assignees
No one assigned
Labels
CI feature request/PR for a new feature NLP
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@michalivne @ericharper @SeanNaren @okuchaiev