[Reland2] Update NVTX to NVTX3 by cyyever · Pull Request #109843 · pytorch/pytorch

cyyever · 2023-09-22T00:47:15Z

Another attempt to update NVTX to NVTX3. We now avoid changing NVTX header inclusion of existing code. The advantage of NVTX3 over NVTX is that it is a header-only library so that linking with NVTX3 can greatly simplify our CMake and other building scripts for finding libraries in user environments. In addition, NVTX are indeed still present in the latest CUDA versions, but they're no longer a compiled library: It's now a header-only library. That's why there isn't a .lib file anymore.

cc @malfet @seemethere @izaitsevfb @peterbell10

pytorch-bot · 2023-09-22T00:47:18Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/109843

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 3 Unrelated Failures

As of commit 8b303bd with merge base 0a25666 ():

NEW FAILURES - The following jobs have failed:

linux-binary-libtorch-cxx11-abi / libtorch-rocm6_0-shared-with-deps-cxx11-abi-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_0-shared-with-deps-pre-cxx11-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_10-rocm6_0-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_11-rocm6_0-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_12-rocm6_0-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_9-rocm6_0-build / build (gh)
ninja: build stopped: subcommand failed

FLAKY - The following job failed but was likely due to flakiness present on trunk:

periodic / win-vs2019-cuda11.8-py3 / test (default, 2, 4, windows.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
test_cuda.py::TestMemPool::test_mempool_with_allocator

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

periodic / linux-focal-rocm6.1-py3.8 / test (distributed, 1, 3, linux.rocm.gpu, unstable) (gh) (#129209)
distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False
periodic / linux-focal-rocm6.1-py3.8 / test (distributed, 2, 3, linux.rocm.gpu, unstable) (gh) (#129209)
distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cyyever · 2023-09-22T00:59:35Z

@izaitsevfb This PR combines your changes in #107497 and more cleanups. And I chose to include 'nvtx3/XXX' to avoid confusion and linking issues between old and newer versions of NVTX.

cyyever · 2023-09-22T01:00:49Z

@pytorchbot label "topic: not user facing"

cyyever · 2023-09-22T01:17:31Z

@pytorchbot label ciflow/binaries

cyyever · 2023-09-22T14:23:32Z

Windows test hosts must install NVTX3 before we can continue. I proposed a PR at pytorch/builder#1547

cyyever · 2023-09-22T23:46:37Z

@pytorchbot rebase

facebook-github-bot · 2024-07-16T21:59:31Z

@izaitsevfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

izaitsevfb · 2024-07-19T17:56:50Z

@izaitsevfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Looks like the testing is passing internally in the current form!

cyyever · 2024-07-19T23:09:47Z

@izaitsevfb Good news..

cyyever · 2024-07-20T04:48:58Z

@izaitsevfb Should I merge it?

izaitsevfb · 2024-07-24T21:45:04Z

@izaitsevfb Should I merge it?

Sorry, I ran a more comprehensive set of builds, and the issue is still there:

xplat/caffe2/torch/csrc/profiler/stubs/cuda.cpp:3:10: fatal error: 'nvtx3/nvToolsExt.h' file not found
#include <nvtx3/nvToolsExt.h>
         ^~~~~~~~~~~~~~~~~~~~

is the old nvtx still used? Why can't we just update it and use the include without the prefix as it was before (#include <nvToolsExt.h>)?

cyyever · 2024-07-25T01:01:16Z

@izaitsevfb Should I merge it?

Sorry, I ran a more comprehensive set of builds, and the issue is still there:
xplat/caffe2/torch/csrc/profiler/stubs/cuda.cpp:3:10: fatal error: 'nvtx3/nvToolsExt.h' file not found
#include <nvtx3/nvToolsExt.h>
         ^~~~~~~~~~~~~~~~~~~~
is the old nvtx still used? Why can't we just update it and use the include without the prefix as it was before (#include <nvToolsExt.h>)?

Now it switches to old nvtx when necessary.

facebook-github-bot · 2024-07-25T17:59:05Z

@izaitsevfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

cyyever · 2024-07-26T00:50:02Z

@pytorchbot rebase

pytorchmergebot · 2024-07-26T00:51:36Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-07-26T00:51:39Z

Successfully rebased nvtx3_fix2 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout nvtx3_fix2 && git pull --rebase)

facebook-github-bot · 2024-07-26T16:16:12Z

@izaitsevfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

izaitsevfb · 2024-07-30T18:13:11Z

@izaitsevfb has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

The latest version passes internal tests.

…ystem

Co-authored-by: Ivan Zaitsev <108101595+izaitsevfb@users.noreply.github.com>

cyyever · 2024-08-20T16:31:42Z

@pytorchmergebot merge -f "Rocm 6.0 was removed"

pytorchmergebot · 2024-08-20T16:33:12Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

cyyever requested review from aaronenyeshi, jeffdaily and jithunnair-amd as code owners September 22, 2023 00:47

cyyever marked this pull request as draft September 22, 2023 00:47

cyyever force-pushed the nvtx3_fix2 branch 4 times, most recently from b2eaaf7 to d260ed6 Compare September 22, 2023 00:57

cyyever changed the title ~~Nvtx3 fix2~~ [Reland2] Upgrade NVTX to NVTX3 Sep 22, 2023

cyyever changed the title ~~[Reland2] Upgrade NVTX to NVTX3~~ [Reland2] Update NVTX to NVTX3 Sep 22, 2023

pytorch-bot Bot added the topic: not user facing topic category label Sep 22, 2023

pytorchbot added the open source label Sep 22, 2023

cyyever force-pushed the nvtx3_fix2 branch from d260ed6 to 8f807d6 Compare September 22, 2023 01:02

cyyever marked this pull request as ready for review September 22, 2023 01:05

cyyever force-pushed the nvtx3_fix2 branch 2 times, most recently from f9d1d49 to eebcc4b Compare September 22, 2023 01:12

pytorch-bot Bot added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Sep 22, 2023

cyyever force-pushed the nvtx3_fix2 branch from eebcc4b to 0aa5697 Compare September 22, 2023 01:18

cyyever marked this pull request as draft September 22, 2023 01:57

cyyever force-pushed the nvtx3_fix2 branch 4 times, most recently from 65f5805 to 52c1190 Compare September 22, 2023 08:25

cyyever marked this pull request as ready for review September 22, 2023 14:23

izaitsevfb reviewed Jul 25, 2024

View reviewed changes

Comment thread torch/utils/hipify/cuda_to_hip_mappings.py Outdated

cyyever and others added 13 commits August 5, 2024 08:53

add more nvtx fixes

6979406

use nvtx3 which is a header only library that can simplify building s…

ec319af

…ystem

Add nvtx3 as third-party library

660e2d2

fix NVTX3 target

536dcef

Use FATAL_ERROR for NVTX3 check

f8e46b7

Cleanup

6a91d63

Switch to old nvtx target

4e3a9c8

Use macro

287221c

Update torch/utils/hipify/cuda_to_hip_mappings.py

57ded7e

Co-authored-by: Ivan Zaitsev <108101595+izaitsevfb@users.noreply.github.com>

Fix nvtx3 header

333ba38

Add lib

4c6b0e7

Fix build errors

c28a58c

Fix build errors

8b303bd

efsotr mentioned this pull request Aug 13, 2024

CMake Error: Failed to find nvToolsExt #133293

Closed

mantaionut mentioned this pull request Aug 23, 2024

Pytorch compile failure on Windows with CUDA 12.1 because of lacking NVTX component #101135

Closed

Conversation

cyyever commented Sep 22, 2023 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/109843

❌ 6 New Failures, 3 Unrelated Failures

Uh oh!

cyyever commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyyever commented Sep 22, 2023

Uh oh!

cyyever commented Sep 22, 2023

Uh oh!

cyyever commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyyever commented Sep 22, 2023

Uh oh!

facebook-github-bot commented Jul 16, 2024

Uh oh!

izaitsevfb commented Jul 19, 2024

Uh oh!

cyyever commented Jul 19, 2024

Uh oh!

cyyever commented Jul 20, 2024

Uh oh!

izaitsevfb commented Jul 24, 2024

Uh oh!

cyyever commented Jul 25, 2024

Uh oh!

facebook-github-bot commented Jul 25, 2024

Uh oh!

Uh oh!

cyyever commented Jul 26, 2024

Uh oh!

pytorchmergebot commented Jul 26, 2024

Uh oh!

pytorchmergebot commented Jul 26, 2024

Uh oh!

facebook-github-bot commented Jul 26, 2024

Uh oh!

izaitsevfb commented Jul 30, 2024

Uh oh!

cyyever commented Aug 20, 2024

Uh oh!

pytorchmergebot commented Aug 20, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

cyyever commented Sep 22, 2023 •

edited by pytorch-bot Bot

Loading

pytorch-bot Bot commented Sep 22, 2023 •

edited

Loading

cyyever commented Sep 22, 2023 •

edited

Loading

cyyever commented Sep 22, 2023 •

edited

Loading