Reapply "Various CUDA graph improvements on capture time, replay time, memory footprint (#2572)" - #3056

Reapply "Various CUDA graph improvements on capture time, replay time, memory footprint (#2572)"#3056

ko3n1g merged 4 commits into

NVIDIA:mainNVIDIA/Megatron-LM:mainfrom

jiemingz:jiemingz/mcore_cudagraph_improvements2jiemingz/Megatron-LM:jiemingz/mcore_cudagraph_improvements2Copy head branch name to clipboard