Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use GPTModel from mcore #7093

Merged
merged 78 commits into from
Aug 14, 2023
Merged

Use GPTModel from mcore #7093

merged 78 commits into from
Aug 14, 2023

Commits on Aug 8, 2023

  1. start adding gpt from megatron core path 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    63c127d View commit details
    Browse the repository at this point in the history

  2. set model parallel config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    16d85c4 View commit details
    Browse the repository at this point in the history

  3. use model parallel config object 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    19e1420 View commit details
    Browse the repository at this point in the history

  4. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a309c0b View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    2ea9285 View commit details
    Browse the repository at this point in the history

  6. set vp size to none if it is 1 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    46ec121 View commit details
    Browse the repository at this point in the history

  7. set vp size to none if it is 1 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    575ef8a View commit details
    Browse the repository at this point in the history

  8. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a8b177c View commit details
    Browse the repository at this point in the history

  9. add TransformerConfig 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a296be8 View commit details
    Browse the repository at this point in the history

  10. start updating to TransformerConfig 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    ec3c170 View commit details
    Browse the repository at this point in the history

  11. add todo 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e2090ae View commit details
    Browse the repository at this point in the history

  12. revert to model parallel config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e1f38d8 View commit details
    Browse the repository at this point in the history

  13. add hidden_size to model_parallel_config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    cbfb0d4 View commit details
    Browse the repository at this point in the history

  14. remove imports 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    cbf5036 View commit details
    Browse the repository at this point in the history

  15. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    c6fe7ed View commit details
    Browse the repository at this point in the history

  16. remove import 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    2bd408c View commit details
    Browse the repository at this point in the history

  17. small clean up 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    06064bf View commit details
    Browse the repository at this point in the history

  18. update hidden size in peft base model, add mcore commit to jenkins 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    d8e9f4f View commit details
    Browse the repository at this point in the history

  19. update module args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    afdf3f0 View commit details
    Browse the repository at this point in the history

  20. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    90e8160 View commit details
    Browse the repository at this point in the history

  21. add config obj to flash attention tests 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    3f194ac View commit details
    Browse the repository at this point in the history

  22. remove args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    95a3b68 View commit details
    Browse the repository at this point in the history

  23. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a0e133f View commit details
    Browse the repository at this point in the history

  24. remove sequence parallel arg 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e2cc69a View commit details
    Browse the repository at this point in the history

  25. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    b4b5b06 View commit details
    Browse the repository at this point in the history

  26. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    bcbd787 View commit details
    Browse the repository at this point in the history

  27. add config to self 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    f31ff9f View commit details
    Browse the repository at this point in the history

  28. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    dd59c1c View commit details
    Browse the repository at this point in the history

  29. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a6d124e View commit details
    Browse the repository at this point in the history

  30. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e36d8c1 View commit details
    Browse the repository at this point in the history

  31. add config to test 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    215b1e5 View commit details
    Browse the repository at this point in the history

  32. get hidden_size from config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    61adaab View commit details
    Browse the repository at this point in the history

  33. add try except 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    8de4993 View commit details
    Browse the repository at this point in the history

  34. use default 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    48942cc View commit details
    Browse the repository at this point in the history

  35. update config with hidden size 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    7a4e884 View commit details
    Browse the repository at this point in the history

  36. remove arg 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    0c008fd View commit details
    Browse the repository at this point in the history

  37. comment out jenkins test 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    148e19a View commit details
    Browse the repository at this point in the history

  38. revert import 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    6d6a69b View commit details
    Browse the repository at this point in the history

  39. remove optimizer_idx 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    bcc6072 View commit details
    Browse the repository at this point in the history

  40. prefetch num microbatches 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    aa5b5fb View commit details
    Browse the repository at this point in the history

  41. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    2db6c2a View commit details
    Browse the repository at this point in the history

  42. Merge branch 'main' into mcore_gpt_path

    @ericharper
    ericharper committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e5d48ae View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. Merge branch 'main' into mcore_gpt_path

    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    6199567 View commit details
    Browse the repository at this point in the history

  2. start adding gpt from megatron core path 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    e5db47e View commit details
    Browse the repository at this point in the history

  3. set model parallel config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    19c95fb View commit details
    Browse the repository at this point in the history

  4. use model parallel config object 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    f7d06e1 View commit details
    Browse the repository at this point in the history

  5. update args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    5c6994e View commit details
    Browse the repository at this point in the history

  6. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    7abc000 View commit details
    Browse the repository at this point in the history

  7. start updating to TransformerConfig 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    eaf75d0 View commit details
    Browse the repository at this point in the history

  8. revert to model parallel config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    55218fc View commit details
    Browse the repository at this point in the history

  9. add hidden_size to model_parallel_config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    52ed0cf View commit details
    Browse the repository at this point in the history

  10. remove imports 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    2ca750a View commit details
    Browse the repository at this point in the history

  11. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    4affce1 View commit details
    Browse the repository at this point in the history

  12. update module args 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    8ae5dec View commit details
    Browse the repository at this point in the history

  13. add config to self 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    3209100 View commit details
    Browse the repository at this point in the history

  14. build transformer config 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    4c7f5cf View commit details
    Browse the repository at this point in the history

  15. add model to provider func 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    73ef19d View commit details
    Browse the repository at this point in the history

  16. update forward and float16 wrapper 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    eb2da31 View commit details
    Browse the repository at this point in the history

  17. instantiate model parallel config after init model parallel 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    83f6101 View commit details
    Browse the repository at this point in the history

  18. set virtual rank 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    6fb875b View commit details
    Browse the repository at this point in the history

  19. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @ericharper
    pre-commit-ci[bot] authored and ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    299e7b5 View commit details
    Browse the repository at this point in the history

  20. Add GQA config to megatron gpt model (#7096) 

    * Add GQA config in gpt config file

Signed-off-by: jasonwan <[email protected]>

* Verify mcore is enabled when using GQA

Signed-off-by: jasonwan <[email protected]>

---------

Signed-off-by: jasonwan <[email protected]>
    @blahBlahhhJ @ericharper
    blahBlahhhJ authored and ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    d4a7576 View commit details
    Browse the repository at this point in the history

  21. revert 

    Signed-off-by: ericharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    3213cac View commit details
    Browse the repository at this point in the history

  22. remove import 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    22de2ef View commit details
    Browse the repository at this point in the history

  23. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    bc38715 View commit details
    Browse the repository at this point in the history

  24. update for dist adam 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    9e7cb71 View commit details
    Browse the repository at this point in the history

  25. Merge branch 'mcore_gpt_model' of github.com:NVIDIA/NeMo into mcore_g… 

    …pt_model
    @ericharper
    ericharper committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    24bcbbe View commit details
    Browse the repository at this point in the history

  26. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    70eeef2 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. use get_gpt_module_list 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    f822996 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mcore_gpt_model' of github.com:NVIDIA/NeMo into mcore_g… 

    …pt_model
    @ericharper
    ericharper committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    a6a10ae View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    2c6ba91 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. pull main 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    5105658 View commit details
    Browse the repository at this point in the history

  2. update megatron core commit 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    9808ec4 View commit details
    Browse the repository at this point in the history

  3. revert change 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    5a20f0d View commit details
    Browse the repository at this point in the history

  4. remove import 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    d5da981 View commit details
    Browse the repository at this point in the history

  5. remove import 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    dbdffc6 View commit details
    Browse the repository at this point in the history

  6. remove import 

    Signed-off-by: eharper <[email protected]>
    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    61a0902 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into mcore_gpt_model

    @ericharper
    ericharper committed Aug 14, 2023
    Configuration menu
    Copy the full SHA
    cb4850d View commit details
    Browse the repository at this point in the history