[mustafabar]

Details

Do not mention proprietary info or link to internal work items in this PR.

Work item: "Internal"

What were the changes?
Support RoCE for MSCCLPP and fix several issues

Why were the changes made?
MSCCLPP did not work with RoCE NICs. Additionally, accidental issues that broke both IB and RoCE were fixed

How was the outcome achieved?

• Auto-detection algorithm for GID index
• MSCCLPP_IB_GID_INDEX for user-level entry of GID index
• Avoid considering NICs that have no active ports (observed issue on local clusters at least)
• Support bootstrapping for Ethernet interfaces with the prefix "en". Without this change, bootstrapping will fail on systems with no IB front NICs (observed issue on local clusters at least)
• Check the successful completion status of WCs. Otherwise, data errors can occur despite receiving a positive number of work completions
• Add hooks for ibv_wc_status_str through the established dynamic loader
• Added AMD peer mem check
• Fix readEnv function which broke gcc 11.4.0 preprocessing when integer specialization is used
• Avoid repeated logic for validating ibv_port
• Accept user-specified TC for RoCE
• Optionally allow using link-local GID (useful for INTRANET)

Additional Documentation:
What else should the reviewer know?

Approval Checklist

Do not approve until these items are satisfied.

• Verify the CHANGELOG has been updated, if
  • there are any NCCL API version changes,
  • any changes impact library users, and/or
  • any changes impact any other ROCm library.

[nileshnegi]

Archiving this PR. Please remove noCI label when ready.

[ammallya]

Cannot import to ROCm/rocm-systems due to merge conflicts please reopen this PR on rocm/rocm-systems.