Inference: Add the embedding and output layer in the full_iteration_inference cuda graph scope for hybrid models - #4440

Inference: Add the embedding and output layer in the full_iteration_inference cuda graph scope for hybrid models#4440

sidsingh-nvidia merged 5 commits into

NVIDIA:mainNVIDIA/Megatron-LM:mainfrom

sidsingh-nvidia:siddharth/cuda-graph-embed-outputsidsingh-nvidia/Megatron-LM:siddharth/cuda-graph-embed-outputCopy head branch name to clipboard