Add MimoOptimizer for heterogeneous parallelism#4019

yashaswikarnati merged 1 commit intoNVIDIA:mainNVIDIA/Megatron-LM:mainfrom

yashaswikarnati:yash/mimo-optimizer-pryashaswikarnati/Megatron-LM:yash/mimo-optimizer-prCopy head branch name to clipboard