[PyTorch][CP] Add THD format support for AllGather-based Context Parallelism#2829

sudhakarsingh27 merged 78 commits into

NVIDIA:mainNVIDIA/TransformerEngine:mainfrom

sudhakarsingh27:cp_thd_swa_with_agsudhakarsingh27/TransformerEngine:cp_thd_swa_with_agCopy head branch name to clipboard