Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restore GPT support for interleaved pipeline parallelism #6528

Merged
merged 18 commits into from
May 9, 2023
Merged

Restore GPT support for interleaved pipeline parallelism #6528

merged 18 commits into from
May 9, 2023

Commits on May 2, 2023

  1. Restore logic for data-parallel communication with pipeline paralleli… 

    …sm in GPT

Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 2, 2023
    Configuration menu
    Copy the full SHA
    8357284 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2023

  1. Support dynamic attention masks in GPT 

    Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 3, 2023
    Configuration menu
    Copy the full SHA
    64b47e4 View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 3, 2023
    Configuration menu
    Copy the full SHA
    c1c1c44 View commit details
    Browse the repository at this point in the history

  3. Debug typos 

    Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 3, 2023
    Configuration menu
    Copy the full SHA
    913b92e View commit details
    Browse the repository at this point in the history

  4. Debug data iterator caching with interleaved pipeline parallelism 

    Each model chunk accesses the data iterator multiple times, so we need to cache multiple samples.

Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 3, 2023
    Configuration menu
    Copy the full SHA
    0a75a74 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 3, 2023
    Configuration menu
    Copy the full SHA
    e537212 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2023

  1. Merge branch 'r1.18.0' into distopt-async-comm 

    Signed-off-by: Eric Harper <[email protected]>
    @ericharper
    ericharper authored May 5, 2023
    Configuration menu
    Copy the full SHA
    4a9516d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'r1.18.0' into distopt-async-comm

    @ericharper
    ericharper authored May 5, 2023
    Configuration menu
    Copy the full SHA
    bb0c90c View commit details
    Browse the repository at this point in the history

Commits on May 8, 2023

  1. Update Megatron-LM commit 

    Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    8e0a8a6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'r1.18.0' into distopt-async-comm

    @timmoon10
    timmoon10 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    b40342b View commit details
    Browse the repository at this point in the history

  3. Distinguish between list of data iterators and data iterator that is … 

    …a list

Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    e1b455f View commit details
    Browse the repository at this point in the history

  4. Create dummy iters to satisy len checks 

    Signed-off-by: Abhinav Khattar <[email protected]>
    @aklife97
    aklife97 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    9725149 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'all_iter' into distopt-async-comm 

    Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 8, 2023
    Configuration menu
    Copy the full SHA
    58b4b7e View commit details
    Browse the repository at this point in the history

Commits on May 9, 2023

  1. Merge branch 'r1.18.0' into distopt-async-comm

    @ericharper
    ericharper authored May 9, 2023
    Configuration menu
    Copy the full SHA
    1c06b8e View commit details
    Browse the repository at this point in the history

  2. Kludge while waiting for Megatron-LM update 

    Signed-off-by: Tim Moon <[email protected]>
    @timmoon10
    timmoon10 committed May 9, 2023
    Configuration menu
    Copy the full SHA
    36ef939 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed May 9, 2023
    Configuration menu
    Copy the full SHA
    c1a4d13 View commit details
    Browse the repository at this point in the history

  4. set transformers offline to avoid rate limiting 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed May 9, 2023
    Configuration menu
    Copy the full SHA
    bd75da7 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'distopt-async-comm' of github.com:timmoon10/NeMo into d… 

    …istopt-async-comm
    @ericharper
    ericharper committed May 9, 2023
    Configuration menu
    Copy the full SHA
    29ad41c View commit details
    Browse the repository at this point in the history