Fix Multi-GPU join for horovod

[irasit]

Horovd's join required local rank as argument for multi-gpu cases, code here
The default join only works for cpu.

Partly fixes #6935
edit: does not seem like any skipped tests were fixed

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

[codecov]

Codecov Report

Merging #6954 (59188cb) into master (b9bc772) will decrease coverage by 5%.
The diff coverage is 10%.

@@           Coverage Diff           @@
##           master   #6954    +/-   ##
=======================================
- Coverage      92%     87%    -5%     
=======================================
  Files         194     194            
  Lines       12329   12333     +4     
=======================================
- Hits        11325   10673   -652     
- Misses       1004    1660   +656     

[awaelchli]

Thanks for the fix <3
I wonder if this solves any of the existing tests that we currently skip. We should try to undo the skip for some of these cpu tests to see if your PR resolves that.
See #6935 for a list.

[irasit]

Thanks for the fix <3
I wonder if this solves any of the existing tests that we currently skip. We should try to undo the skip for some of these cpu tests to see if your PR resolves that.
See #6935 for a list.

Thank you for the reply. I do not think it related to the skipped tests, but seems related to this one, wonder why the issue was not caught by this test before.
The issue is reproducible when I run my test with multiple gpu (not reproducible if only run in cpu). In this test, petastorm reader and spark estimator is used.

[Borda]

shall we reenable Horovod tests back?

[tchaton]

LGTM ! Thanks for your contribution.