Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT REVIEW] debug float8 all-gather numerics #873

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

[DO NOT REVIEW] debug float8 all-gather numerics #873

wants to merge 27 commits into from

Commits on Sep 11, 2024

  1. debug numerics 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    e17155c View commit details
    Browse the repository at this point in the history

  2. print all mismatch 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    544a09c View commit details
    Browse the repository at this point in the history

  3. init linears 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    dd357d6 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. repro with single linear 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    a880ec2 View commit details
    Browse the repository at this point in the history

  2. bitwise equal 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    1ff11c9 View commit details
    Browse the repository at this point in the history

  3. bitwise equal 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    2059918 View commit details
    Browse the repository at this point in the history

  4. remove change on orignal test 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    e8385b2 View commit details
    Browse the repository at this point in the history

  5. restore 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8109bbe View commit details
    Browse the repository at this point in the history

  6. restore 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    72f8d7e View commit details
    Browse the repository at this point in the history

  7. clean repro 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    760dff9 View commit details
    Browse the repository at this point in the history

  8. make baseline FSDP too 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    c39d2d8 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. delayed linear are on par 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    a3baf9f View commit details
    Browse the repository at this point in the history

  2. remove uncessary change 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    095a2c5 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2024

  1. numeric on par with float32 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    a263329 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. add fully_shard to float32 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    653444c View commit details
    Browse the repository at this point in the history

  2. compile float8Linear on-par 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0a4b91d View commit details
    Browse the repository at this point in the history

  3. compile + transformer root works 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    fc22626 View commit details
    Browse the repository at this point in the history

  4. bfloat16 works fine for torch.compile + float8linear, basic root 

    transformer
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    05f467e View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. float8 with/o precompute + bf16/fp32 on par on single float8liner and 

    simple transformer root
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c584dca View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. bf16/fp32 + float8linear bitwise equal, precompute=False 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    569f862 View commit details
    Browse the repository at this point in the history

  2. bf16/fp32 parity for full transformer, precompute=False 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    003dfc3 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. float8linear bitwise on par between eager and compile 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    37fb7a5 View commit details
    Browse the repository at this point in the history

  2. eager is on par with float64 numerics. fixing torch.compile 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    8450886 View commit details
    Browse the repository at this point in the history

  3. trying to replicate bfloat16 in test_base unit test 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f4688ae View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. test_base numerics on reciprocal 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    432230a View commit details
    Browse the repository at this point in the history

  2. _data parity 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    78b8a8d View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. transformer on par becuase of upcasting to float64 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    fc6063c View commit details
    Browse the repository at this point in the history