Repository navigation
test(rccl): cover every gin_host.cc entry point with host-only tests #13225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
e235402
[AICOMRCCL-2642] [rccl] Cover every gin_host.cc entry point with host…
mch 5a728ca
[AICOMRCCL-2642] [rccl] Move gin-host-test fakes onto their owning fa…
mch 2bee9cb
[AICOMRCCL-2642] [rccl] Promote reusable gin-host-test utilities to s…
mch 6ecbf53
[AICOMRCCL-2642] [rccl] Pin the contracts gin-host-test's thin tests …
mch 236091a
[AICOMRCCL-2642] [rccl] Drive the gin-host devComm setup tests throug…
mch 261597c
[AICOMRCCL-2642] [rccl] Name the gin-host microtests after the behavi…
mch 7e2749b
[AICOMRCCL-2642] [rccl] Give FakeGin tagged handles instead of a file…
mch 1424d45
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch e71d114
[AICOMRCCL-2642] [rccl] Follow ncclSocketToString's new buffer-length…
mch 807d2a3
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch f249eb4
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch 4ceee3a
[AICOMRCCL-2642] [rccl] Drop duplicate ncclSetThreadName stub in rma-…
mch 70b8e91
[AICOMRCCL-2642] [rccl] Move ncclTopoGetLocalGinDevs onto fakes/topo_…
mch 3b3e25e
[AICOMRCCL-2642] [rccl] Make the os_fakes affinity recorders safe to …
mch 0f8108e
[AICOMRCCL-2642] [rccl] Use an in-range ginCommCount poison
mch 55d89f5
[AICOMRCCL-2642] [rccl] Trim the host-test helper comments
mch 7723c90
[AICOMRCCL-2642] [rccl] Assert the failing progress thread actually r…
mch 451bb55
[AICOMRCCL-2642] [rccl] Cover GIN_NCONNECTIONS on its own
mch 1a9ad14
[AICOMRCCL-2642] [rccl] Assert the handle array the rail-only team co…
mch 06e1557
[AICOMRCCL-2642] [rccl] Assert which device each GIN connection is op…
mch c73c279
[AICOMRCCL-2642] [rccl] Assert the stride reciprocal and device types…
mch 62c5e77
[AICOMRCCL-2642] [rccl] Cover deregistration across two active backends
mch 198b0af
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch 8e56909
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch a46fffe
Merge branch 'develop' into users/mahughes/aicomrccl-2642-gin-host-tests
mch de98a72
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch c66f9f8
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch f7024a7
[AICOMRCCL-2642] [rccl] Reset HIP fakes after dev-runtime fakes in Co…
mch ecb9c36
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch 51df229
Merge remote-tracking branch 'origin/develop' into users/mahughes/aic…
mch 60291ba
Merge branch 'develop' into users/mahughes/aicomrccl-2642-gin-host-tests
mch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| /************************************************************************* | ||
| * Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. | ||
| * | ||
| * See LICENSE.txt for license information | ||
| ************************************************************************/ | ||
|
|
||
| #ifndef RCCL_TEST_HOST_JOIN_THREADS_ON_EXIT_H_ | ||
| #define RCCL_TEST_HOST_JOIN_THREADS_ON_EXIT_H_ | ||
|
|
||
| #include <functional> | ||
| #include <thread> | ||
| #include <utility> | ||
| #include <vector> | ||
|
|
||
| // JoinThreadsOnExit -- spawn worker threads, stop and join them on scope exit | ||
| // | ||
| // Destroying a still-joinable std::thread calls std::terminate(), so a test that | ||
| // spawns a worker and then trips a fatal `ASSERT_*` would abort the whole binary | ||
| // rather than fail one test. Spawning through this type means the stop-and-join | ||
| // happens on every exit path, including that one. | ||
| // | ||
| // The loop a worker runs is usually ended by a flag the unit under test owns | ||
| // rather than by the thread returning on its own, so the stop action is a | ||
| // callback supplied by the caller -- this type knows nothing about what makes | ||
| // the workers finish, only that it must be done before joining them. Pass no | ||
| // callback for workers that finish by themselves. | ||
| // | ||
| // Usage: | ||
| // | ||
| // JoinThreadsOnExit workers([gs] { gs->stopSignal.store(true); }); | ||
| // workers.spawn([gs] { ncclGinProgress(gs, 0); }); | ||
| // ASSERT_TRUE(waitUntil([&] { return fake_.progressCalls.load() > 0; })); | ||
| // // workers' destructor raises the stop signal and joins, even on the ASSERT. | ||
| // | ||
| // Non-copyable and non-movable so the threads have exactly one owner; C++17 | ||
| // guaranteed copy elision still allows a factory that binds the stop callback to | ||
| // return one by value (`auto workers = progressWorkers(gs);`). | ||
| class JoinThreadsOnExit { | ||
| public: | ||
| explicit JoinThreadsOnExit(std::function<void()> stop = {}) : stop_(std::move(stop)) {} | ||
| ~JoinThreadsOnExit() { stopAndJoin(); } | ||
|
|
||
| JoinThreadsOnExit(const JoinThreadsOnExit&) = delete; | ||
| JoinThreadsOnExit& operator=(const JoinThreadsOnExit&) = delete; | ||
| JoinThreadsOnExit(JoinThreadsOnExit&&) = delete; | ||
| JoinThreadsOnExit& operator=(JoinThreadsOnExit&&) = delete; | ||
|
|
||
| template <class Fn> | ||
| void spawn(Fn&& fn) { threads_.emplace_back(std::forward<Fn>(fn)); } | ||
|
|
||
| // Also callable before the scope ends, for a test that asserts on what the | ||
| // workers left behind once they have all stopped. Idempotent. | ||
| void stopAndJoin() | ||
| { | ||
| if (threads_.empty()) return; | ||
| if (stop_) stop_(); | ||
| for (auto& t : threads_) { | ||
| if (t.joinable()) t.join(); | ||
| } | ||
| threads_.clear(); | ||
| } | ||
|
|
||
| private: | ||
| std::function<void()> stop_; | ||
| std::vector<std::thread> threads_; | ||
| }; | ||
|
|
||
| #endif // RCCL_TEST_HOST_JOIN_THREADS_ON_EXIT_H_ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| /************************************************************************* | ||
| * Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. | ||
| * | ||
| * See LICENSE.txt for license information | ||
| ************************************************************************/ | ||
|
|
||
| #ifndef RCCL_TEST_HOST_SCOPE_EXIT_H_ | ||
| #define RCCL_TEST_HOST_SCOPE_EXIT_H_ | ||
|
|
||
| #include <utility> | ||
|
|
||
| // ScopeExit -- run this cleanup on every exit from the scope | ||
| // | ||
| // A fatal assertion (`ASSERT_*`) returns from the test body immediately, so | ||
| // cleanup written after the assertion simply does not run -- which in a host | ||
| // test usually means a leaked allocation, a still-joinable std::thread (that | ||
| // calls std::terminate from its destructor), or a fake left pointing at a dead | ||
| // stack local, so one failing test takes the rest of the binary with it. Put | ||
| // that cleanup in a ScopeExit at the point the resource is acquired and it runs | ||
| // on the fatal path too. | ||
| // | ||
| // Usage (CTAD deduces the callable's type): | ||
| // | ||
| // std::thread worker([&] { ... }); | ||
| // ScopeExit joinWorker([&] { stop.store(true); worker.join(); }); | ||
| // ASSERT_TRUE(waitUntil([&] { return started.load(); })); // safe to bail here | ||
| // | ||
| // Prefer a purpose-built RAII type where one already exists (ScopedHook for a | ||
| // std::function seam, JoinThreadsOnExit for a set of worker threads); ScopeExit | ||
| // is for the one-off cleanup that does not have one. | ||
| // | ||
| // Non-copyable and non-movable so the cleanup cannot run twice; C++17 guaranteed | ||
| // copy elision still allows a factory to return one by value. | ||
| template <typename Fn> | ||
| class ScopeExit { | ||
| public: | ||
| explicit ScopeExit(Fn fn) : fn_(std::move(fn)) {} | ||
| ~ScopeExit() { fn_(); } | ||
|
|
||
| ScopeExit(const ScopeExit&) = delete; | ||
| ScopeExit& operator=(const ScopeExit&) = delete; | ||
| ScopeExit(ScopeExit&&) = delete; | ||
| ScopeExit& operator=(ScopeExit&&) = delete; | ||
|
|
||
| private: | ||
| Fn fn_; | ||
| }; | ||
|
|
||
| template <typename Fn> | ||
| ScopeExit(Fn) -> ScopeExit<Fn>; | ||
|
|
||
| #endif // RCCL_TEST_HOST_SCOPE_EXIT_H_ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| /************************************************************************* | ||
| * Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. | ||
| * | ||
| * See LICENSE.txt for license information | ||
| ************************************************************************/ | ||
|
|
||
| #ifndef RCCL_TEST_HOST_SCRIPTED_FAILURE_H_ | ||
| #define RCCL_TEST_HOST_SCRIPTED_FAILURE_H_ | ||
|
|
||
| #include "nccl.h" | ||
|
|
||
| // ScriptedFailure -- "fail call N of this entry point" | ||
| // | ||
| // A test that needs the *second* connect() to fail otherwise grows a bespoke | ||
| // pair of file-scope knobs per seam (`g_fooCalls` + `g_failFooCall`) and a | ||
| // hand-written comparison inside every fake. ScriptedFailure is that pattern | ||
| // once: the fake asks what to return for this call number, and the test says | ||
| // which call fails with what. | ||
| // | ||
| // `onCall` is 1-based; 0 means "every call", so a test that only cares that the | ||
| // seam fails does not have to know how many times production calls it. | ||
| // | ||
| // Usage -- in the fake, next to the counter the test asserts on: | ||
| // | ||
| // int connectCalls = 0; | ||
| // ScriptedFailure failConnect; // default: every call succeeds | ||
| // | ||
| // ncclResult_t Connect(...) { | ||
| // ncclResult_t ret = failConnect.at(++connectCalls); | ||
| // if (ret != ncclSuccess) return ret; | ||
| // ...succeed... | ||
| // } | ||
| // | ||
| // and in the test: | ||
| // | ||
| // fake_.failConnect = {ncclSystemError, 2}; // the second connect fails | ||
| // | ||
| // Deliberately a plain aggregate with no counter of its own: tests assign whole | ||
| // scripts with `= {ncclSystemError, 2}` and reset them with `= {}`, and the call | ||
| // counter usually has to be visible to the test's own assertions anyway. Keeping | ||
| // the count outside also leaves the fake free to share one script across several | ||
| // entry points, or to count calls the script does not care about. | ||
| struct ScriptedFailure { | ||
| ncclResult_t result = ncclSuccess; | ||
| int onCall = 0; | ||
|
|
||
| // Status the seam should return for call number `callNumber` (1-based). | ||
| ncclResult_t at(int callNumber) const | ||
| { | ||
| if (result == ncclSuccess) return ncclSuccess; | ||
| if (onCall == 0 || onCall == callNumber) return result; | ||
| return ncclSuccess; | ||
| } | ||
| }; | ||
|
|
||
| #endif // RCCL_TEST_HOST_SCRIPTED_FAILURE_H_ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| /************************************************************************* | ||
| * Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. | ||
| * | ||
| * See LICENSE.txt for license information | ||
| ************************************************************************/ | ||
|
|
||
| #ifndef RCCL_TEST_HOST_WAIT_UNTIL_H_ | ||
| #define RCCL_TEST_HOST_WAIT_UNTIL_H_ | ||
|
|
||
| #include <chrono> | ||
| #include <thread> | ||
|
|
||
| // waitUntil -- bounded poll for something another thread will do | ||
| // | ||
| // A test that spawns a worker cannot assert on the worker's effect immediately, | ||
| // and must not sleep-and-hope either: a fixed sleep is both slower than it needs | ||
| // to be on a good run and still flaky on a loaded machine. waitUntil returns as | ||
| // soon as `pred()` holds, and returns false once `budget` has elapsed so the | ||
| // test fails with its own message instead of hanging until the suite's timeout: | ||
| // | ||
| // ASSERT_TRUE(waitUntil([&] { return fake_.progressCalls.load() > 0; })) | ||
| // << "the progress thread never called ginProgress"; | ||
| // | ||
| // The default budget is deliberately generous: it is never waited for on a | ||
| // passing run (the predicate ends the wait), and it only has to be long enough | ||
| // that a heavily loaded CI machine does not trip it. | ||
| inline constexpr auto kWaitUntilBudget = std::chrono::milliseconds(2000); | ||
|
|
||
| template <typename Pred> | ||
| bool waitUntil(Pred pred, std::chrono::milliseconds budget = kWaitUntilBudget) | ||
| { | ||
| const auto deadline = std::chrono::steady_clock::now() + budget; | ||
| while (!pred()) { | ||
| if (std::chrono::steady_clock::now() >= deadline) return false; | ||
| std::this_thread::yield(); | ||
| } | ||
| return true; | ||
| } | ||
|
|
||
| #endif // RCCL_TEST_HOST_WAIT_UNTIL_H_ |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.