[DO NOT REVIEW] debug float8 all-gather numerics

[weifengpy]

repro numeric difference with single Float8Linear module with/without float8 all-gather

repro: TORCH_LOGS="output_code" pytest -s test/float8/test_fsdp2/test_fsdp2.py -k test_float8_linear_parity 2>&1 | tee test_float8_linear_parity.log

patch pytorch with following

diff --git a/torch/distributed/_composable/fsdp/_fsdp_collectives.py b/torch/distributed/_composable/fsdp/_fsdp_collectives.py
index 4e10f4594c..150752b2f5 100644
--- a/torch/distributed/_composable/fsdp/_fsdp_collectives.py
+++ b/torch/distributed/_composable/fsdp/_fsdp_collectives.py
@@ -442,6 +442,8 @@ def _get_all_gather_input_metadatas(
             input_numels.append(all_gather_input.numel())
         param_all_gather_input_dtypes.append(input_dtypes)
         param_all_gather_input_numels.append(input_numels)
+    if all_gather_dtype == torch.float8_e4m3fn:
+        all_gather_dtype = torch.uint8
     return (
         param_all_gather_input_dtypes,
         param_all_gather_input_numels,

P1585298991

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/873

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 10 New Failures

As of commit fc6063c with merge base 85d03de ():

NEW FAILURES - The following jobs have failed:

• Code Analysis with Ruff / build (3.9) (gh)
• Run Float8 Tests / test (SM-89, linux.g6.4xlarge.experimental.nvidia.gpu, --pre torch --index-url https://download.p... / linux-job (gh)
  RuntimeError: Command docker exec -t ce9e0792c756a98c359ddb92a11215dede6838f2a5799d76d55ffbed78816955 /exec failed with exit code 2
• Run Regression Tests / test (CPU 2.2.2, linux.4xlarge, torch==2.2.2 --index-url https://download.pytorch.org/whl/cpu "nu... / linux-job (gh)
  RuntimeError: Command docker exec -t 553a3cec8a461b3ddba7f758138814473479954c294a501e002a940b98413061 /exec failed with exit code 2
• Run Regression Tests / test (CPU 2.3, linux.4xlarge, torch==2.3.0 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t 1eb485c1cec03f47c2a467ff3a0b8e19465762ea11c39015195910094802594a /exec failed with exit code 2
• Run Regression Tests / test (CPU 2.4, linux.4xlarge, torch==2.4.0 --index-url https://download.pytorch.org/whl/cpu, cpu) / linux-job (gh)
  RuntimeError: Command docker exec -t 13250ace0fd57e6e1a6e7959773558b7859e687739918bf186338af7506cdfb3 /exec failed with exit code 2
• Run Regression Tests / test (CPU Nightly, linux.4xlarge, --pre torch --index-url https://download.pytorch.org/whl/nightl... / linux-job (gh)
  RuntimeError: Command docker exec -t e67470e83b534bbd7b6f94304568261f35a1f9360bdb8df45426c1495564cb1a /exec failed with exit code 2
• Run Regression Tests / test (CUDA 2.2.2, linux.g5.12xlarge.nvidia.gpu, torch==2.2.2 "numpy<2" , cuda, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t de180a7b5095261315efc56a2e56d247fff6e6f4da265105d85e00c992d6a512 /exec failed with exit code 2
• Run Regression Tests / test (CUDA 2.3, linux.g5.12xlarge.nvidia.gpu, torch==2.3.0, cuda, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 3ed8fec50c7f015a5069d90596cc946b9bea4c9ffbc9c757fe31440d6de8e417 /exec failed with exit code 2
• Run Regression Tests / test (CUDA 2.4, linux.g5.12xlarge.nvidia.gpu, torch==2.4.0, cuda, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e5b8560e693263bd09f7a1f115dc628c9a11cc6cdabde54b4e082109d73f243 /exec failed with exit code 2
• Run Regression Tests / test (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://download.pytorc... / linux-job (gh)
  RuntimeError: Command docker exec -t 3ef1af5f19a7b8d46712180e18125410c23ab7bd072e00ea33e3ef02a26afb55 /exec failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.