UCP: Name the peer version checks by dmitrygx · Pull Request #12233 · openucx/ucx

dmitrygx · 2026-11-04T08:21:49Z

What?

Name the peer release versions that UCP compares dst_version with. One commented enum of UCP_MIN_DST_VERSION_* constants in core/ucp_types.h replaces the raw numbers and the scattered defines, and the UCP agent guides ask to gate such checks by these constants. No functional change.

Why?

A raw release number does not say which peer capability it gates, and a new check can easily break compatibility with older peers. This came up in the review of #12206.

How?

The constants keep the old values: checks like > 19 become >= 20 (dst_version is unsigned). The enum also takes the existing constants from wireup.h and rma_rndv.c, renamed to the same prefix, and UCP_RELEASE_LEGACY moves next to it as UCP_VERSION_LEGACY. Includes the changes of the closed #12234.

select.c compared dst_version of the remote address with the raw release numbers 17, 18 and 19. Define them as UCP_WIREUP_*_MIN_DST_VERSION constants in wireup.h, next to UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION, so that the peer release checks of the wireup are in one place. The values are not changed.

A change of the lanes picked by select.c must keep the old selection with peers of older releases, otherwise a new and an old peer pick different lanes. Document that such a change is gated by a *_MIN_DST_VERSION constant in wireup.h.

svc-nvidia-pr-review · 2026-11-04T08:22:56Z

🤖 Starting review — findings will be posted here when done.

ucp_rkey.c, ucp_ep.c and am/eager_multi.c compared dst_version of the peer with raw release numbers. Define them as local *_MIN_DST_VERSION constants, like UCP_PROTO_RMA_RNDV_MIN_DST_VERSION in rma_rndv.c: - UCP_RKEY_SYS_DEV_MIN_DST_VERSION (20): the peer always packs the system device of a non-empty remote key. "> 19" becomes ">= 20", dst_version is unsigned. - UCP_EP_RECOVERY_MIN_DST_VERSION (22): the peer supports the endpoint recovery. The diagnostic message prints it instead of a literal 1.22. - UCP_AM_PSN_MIN_DST_VERSION (21): the peer supports the PSN protocol. No functional change.

The peer release is checked not only by the lane selection, but also by the remote key unpack, the endpoint recovery and the protocols. Move the rule from wireup/AGENTS.md to src/ucp/AGENTS.md and make it cover any code that depends on what the peer release supports.

svc-nvidia-pr-review · 2026-11-04T08:56:14Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-04T09:00:47Z

+  checked against `dst_version` of the peer, never by a raw release
+  number. Define the constant with a comment in the module that checks
+  it, e.g. `UCP_WIREUP_MAX_LANES_MIN_DST_VERSION` in `wireup/wireup.h` or
+  `UCP_AM_PSN_MIN_DST_VERSION` in `am/eager_multi.c`.


src/ucp/wireup/address.c:1642 (not in the diff, surfaced by this new rule): ucp_address_adjust_unpacked_md_index() still gates peer-dependent unpacking on a raw release number via a local const unsigned max_legacy_api_version = 18u;. The PR documents "never by a raw release number", so this is the one leftover that contradicts the new rule. Can we convert it to a *_MIN_DST_VERSION constant as well, so the new rule holds everywhere?

Done, UCP_ADDRESS_MD_INDEX_MIN_DST_VERSION.

svc-nvidia-pr-review · 2026-11-04T09:00:47Z


+/* Minimal peer release version which supports the endpoint recovery */
+#define UCP_EP_RECOVERY_MIN_DST_VERSION 22
+


minor: the new define is separated by a single blank line from __KHASH_IMPL, while the two other defines added in this PR (am/eager_multi.c, core/ucp_rkey.c) use two blank lines. Pls add one more blank line before __KHASH_IMPL to match.

No define is left in ucp_ep.c: the constants moved to one enum in ucp_types.h.

evgeny-leksikov · 2026-11-04T09:21:02Z

 #include <string.h>

+/* Minimal peer release version which supports the endpoint recovery */
+#define UCP_EP_RECOVERY_MIN_DST_VERSION 22


maybe move the list of constants to single place/enum ?

Done: all the *_MIN_DST_VERSION constants are now one enum in ucp_types.h, ordered by version.

Keep all the *_MIN_DST_VERSION constants, including the existing UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION and UCP_PROTO_RMA_RNDV_MIN_DST_VERSION, in one enum in ucp_types.h, ordered by version, instead of next to their users. The AGENTS.md rule points to this list.

ucp_address_adjust_unpacked_md_index() compared dst_version with a local constant 18, the last release which packs legacy bits in the MD index field of a v1 address. Use UCP_ADDRESS_MD_INDEX_MIN_DST_VERSION (19) instead: "> 18" becomes ">= 19", dst_version is unsigned.

svc-nvidia-pr-review · 2026-11-04T11:58:44Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-04T12:03:43Z

@@ -60,6 +60,12 @@ applications. Almost every meaningful UCX feature lives here.
 - For data sent during wireup (worker addresses, capability bits, etc.),


minor: can we also update wireup/AGENTS.md ("A payload section appended to an already released message must be gated by a *_MIN_DST_VERSION constant ...") to say the constant goes to the list in core/ucp_types.h? Now that UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION moved out of wireup.h, the wireup guide still describes *_MIN_DST_VERSION gating locally without saying where the constant belongs — otherwise the next wire-compat constant may again be defined locally in wireup.h.

Done: the wireup guide now takes the constant from core/ucp_types.h.

svc-nvidia-pr-review · 2026-11-04T12:03:43Z

@@ -349,6 +349,36 @@ typedef enum {
 } ucp_object_version_t;


Residual note: two version gates stay outside the new list by design (select.c:1527, select.c:2302, dst_version > UCP_RELEASE_LEGACY). If the intent is that all peer-release gating is discoverable from ucp_types.h, it may be worth a one-line comment here. Not raised as a blocking comment since behavior is unchanged and the constant is already symbolic.

Added a sentence to the comment of the list: a peer which does not pack its release version has dst_version UCP_RELEASE_LEGACY, below all of the listed versions. The two > UCP_RELEASE_LEGACY checks stay as they are: they check whether the peer packs its release version at all (UCX 1.16 and later), and the constant is already named.

The wireup guide asks to gate a payload section appended to a released message by a *_MIN_DST_VERSION constant. Say that the constant comes from core/ucp_types.h, which now has UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION too, so that a new one is not defined in wireup.h again.

A peer which does not pack its release version has dst_version UCP_RELEASE_LEGACY. Say so in the comment of the *_MIN_DST_VERSION list: such a peer is below all of the versions there, and select.c checks dst_version > UCP_RELEASE_LEGACY to know whether the peer packs its release version at all.

svc-nvidia-pr-review · 2026-11-04T12:33:58Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-04T12:37:54Z

+ * A feature is used with a peer only if the release version of the peer
+ * (dst_version) is at least the given one, so peers of older releases keep
+ * working. A peer which does not pack its release version has dst_version
+ * UCP_RELEASE_LEGACY, below all of these versions.


minor: UCP_RELEASE_LEGACY, which this comment refers to, is still defined in wireup/wireup.h — can we move it here too, so all peer-version constants live in one place?

Done: UCP_RELEASE_LEGACY is now defined in core/ucp_types.h, right before the list.

UCP_RELEASE_LEGACY is the dst_version of a peer which does not pack its release version, and the comment of the *_MIN_DST_VERSION list refers to it. Define it in core/ucp_types.h next to the list instead of wireup/wireup.h, so all the peer version constants are in one place. No functional change.

svc-nvidia-pr-review · 2026-11-04T13:05:19Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-04T13:10:59Z

 } ucp_object_version_t;


+/* The dst_version of a peer which does not pack its release version */


minor: maybe make this the first member of the enum below (UCP_RELEASE_LEGACY = 0), so all peer-version constants are defined in one list?

Done: UCP_RELEASE_LEGACY = 0 is now the first member of the list.

svc-nvidia-pr-review · 2026-11-04T13:10:59Z

+    UCP_PROTO_RMA_RNDV_MIN_DST_VERSION = 22,
+    /* The token trailer of the LANES_ADDR messages */
+    UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION = 24
+};


can we add UCS_STATIC_ASSERT(UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION <= UCP_API_MINOR) here? Since the list is ordered, asserting the last one would catch a typo that silently disables a feature even between same-release peers.

UCS_STATIC_ASSERT is a switch statement, so it can only go inside a function, not next to the list, and asserting the last constant would have to be moved with every new one. I'd keep the list as it is.

svc-nvidia-pr-review · 2026-11-04T13:10:59Z

-    if (ucp_ep_config(ep)->key.dst_version < 22) {
-        ucs_diag("ep: %p: recovery support requires UCX 1.22 or later, "
+    if (ucp_ep_config(ep)->key.dst_version < UCP_EP_RECOVERY_MIN_DST_VERSION) {
+        ucs_diag("ep: %p: recovery support requires UCX 1.%d or later, "


Suggested change

ucs_diag("ep: %p: recovery support requires UCX 1.%d or later, "

ucs_diag("ep: %p: recovery support requires UCX 1.%d or later, "

"remote peer version %u is not supported",

ep, UCP_EP_RECOVERY_MIN_DST_VERSION,

ucp_ep_config(ep)->key.dst_version);

minor: dst_version is unsigned.

Done, %u.

Keep all the peer version constants in one enum: UCP_RELEASE_LEGACY, the dst_version of a peer which does not pack its release version, becomes its first member instead of a define next to it. The list is ordered by version, so the comment no longer has to say that UCP_RELEASE_LEGACY is below all the other versions. No functional change.

dst_version is unsigned.

svc-nvidia-pr-review · 2026-11-04T13:18:18Z

🤖 Starting review — findings will be posted here when done.

dmitrygx · 2026-11-04T15:09:13Z

@evgeny-leksikov @guy-ealey-morag could you review pls? I guess now your comments fully addressed.

guy-ealey-morag · 2026-11-04T15:15:48Z

+ * used with a peer only if dst_version of the peer is at least the given one,
+ * so peers of older releases keep working.
+ */
+enum {


minor: as an enum I would expect the values to have the same prefix like UCP_MIN_DST_VERSION_*
It's not super important so your call

+1 for renamimg

Done: all of them are UCP_MIN_DST_VERSION_* now, UCP_RELEASE_LEGACY became UCP_MIN_DST_VERSION_LEGACY.

The constants are members of one enum, so give them a common prefix: UCP_<FEATURE>_MIN_DST_VERSION becomes UCP_MIN_DST_VERSION_<FEATURE>, and UCP_RELEASE_LEGACY becomes UCP_MIN_DST_VERSION_LEGACY. No functional change.

The code style check aligns the values of consecutive enum members, also across the comments between them.

svc-nvidia-pr-review · 2026-11-04T19:22:00Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-04T19:26:24Z

+ */
+enum {
+    /* The dst_version of a peer which does not pack its release version */
+    UCP_MIN_DST_VERSION_LEGACY                    = 0,


IMO this entry is not a feature minimum - both users check dst_version > UCP_MIN_DST_VERSION_LEGACY - so UCP_DST_VERSION_LEGACY would read better and match the comment above.

I'd keep it: the reviewers asked for one prefix for all the values of this enum, and as its lowest value UCP_MIN_DST_VERSION_LEGACY is the minimal dst_version, that of a peer which does not pack its release version.

svc-nvidia-pr-review · 2026-11-04T19:26:24Z

+ */
+enum {
+    /* The dst_version of a peer which does not pack its release version */
+    UCP_MIN_DST_VERSION_LEGACY                    = 0,


minor: ucp_address_unpack_header() still does *dst_version = 0 for a peer that does not pack its release version, can we use this constant there too?

ucp_address_unpack_header() sets dst_version of a v1 address to 0 until the release version is found after the devices, as for a peer which does not pack it. Use the named constant. No functional change.

The release version is packed in 4 bits, and ucp_address_unpack_release_version() takes the value closest to our release, so a peer more than 7 releases older can get a wrong, newer dst_version (with release 24, a 1.16 peer is seen as 32). Say so in the comment of the list.

svc-nvidia-pr-review · 2026-11-05T07:36:37Z

🤖 Starting review — findings will be posted here when done.

UCP_MIN_DST_VERSION_LEGACY is not the minimal peer version of a feature: it is the dst_version of a peer which does not pack its release version, and select.c only checks that dst_version is above it. Define it as UCP_VERSION_LEGACY next to the enum instead. No functional change.

svc-nvidia-pr-review · 2026-11-05T15:27:09Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-05T15:31:59Z



+/* The dst_version of a peer which does not pack its release version */
+#define UCP_VERSION_LEGACY 0


minor: can we call it UCP_DST_VERSION_LEGACY and move it into the enum below? Its own comment says it is a dst_version value, so it would be good to keep all of them in one place with the same prefix. As it stands, the renamed legacy constant is the only dst_version constant that neither carries the DST_VERSION prefix nor lives in the new enum, so the grouping this PR introduces is slightly inconsistent.

svc-ucx · 2026-11-05T17:13:23Z

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 1) · commit ac3d55a5

TL;DR: The ASAN/BlueField gtest shard 2/4 hung for ~11 minutes in dcx/test_ucp_rma.put_nonblocking_iov_zcopy/2 <dc_x,cuda_copy,rocm_copy/user_memh> — the ucp_ep_flush_nbx() issued after a multi-lane IOV put-zcopy never completed, so gtest hit its deadline and aborted (core dump → make test exit 2). This is a real hang in the UCP multi-rail SGL/IOV put-zcopy + selective EP-flush path, not something PR #12233 (a pure rename of the peer-version constants) can cause.

Full analysis

Summary: make -C test/gtest test aborted because test_ucp_rma.put_nonblocking_iov_zcopy (dc_x, user_memh variant) deadlocked waiting for an endpoint flush to complete.

Root cause: Log timeline shows a hang, not slowness:

16:47:26.1 — [ RUN ] dcx/test_ucp_rma.put_nonblocking_iov_zcopy/2 <dc_x,cuda_copy,rocm_copy/user_memh>
11 min 16 s of total silence (largest gap in the whole job)
16:58:42.5 — ucp_test.cc:310 UCX ERROR request 0xffff45b0a2e0 did not complete on time
17:02:26 — test_helpers.cc:57: Failure / Connection timed out - abort testing → Caught signal 6 (Aborted)

The backtrace pins the stuck request precisely: test_ucp_rma::do_nbi_iov() (test_ucp_rma.cc:56) → ucp_test::flush_ep() (ucp_test.cc:203) → request_process() spinning in ucp_test::progress(). So the put was posted and the subsequent per-EP flush request stayed UCS_INPROGRESS forever. The test configures MAX_RMA_LANES=2 (test_ucp_rma.cc:39) and runs the USER_MEMH variant over dc_mlx5, i.e. exactly the new multi-lane IOV/SGL put-zcopy + remote-flush path.

Prime suspect in that path: ucp_proto_put_offload_zcopy_send_func() registers a remote-flush entry for every posted fragment (ucp_proto_put_offload_update_remote_flush, put_offload.c:282-285), and ucp_ep_flush_mem_start() (flush.c:278-328) then clears ep->ext->flush_sys_dev_map, sets mem.uct_comp.count = count and returns UCS_INPROGRESS — but ucp_ep_flush_mem_completion() (flush.c:267-276) only asserts and traces; it never invokes req->send.flushed_cb/completes the flush request. If the mem-flush completion is the last step, the user flush request can never complete, which matches the symptom exactly. Note also the test's first loop iteration posts a zero-length IOV put (iov_count = 0, test_ucp_rma.cc:52-55) which still registers a remote-flush entry.

Nothing in the PR under test touches RMA/flush: commit 76759f5a / ac3d55a5 only prefix the peer-version constants (UCP_MIN_DST_VERSION_*), and in gtest both peers are the same build, so version gating is symmetric and cannot change protocol selection.

Implicated commit: not the PR commit ac3d55a5. Candidates, in order of likelihood: d513f98f (nbellalou, "UCP/FLUSH: Add topology-safe selective endpoint flush", #11923), 4af0b6b5 (Michal Shalev, "UCP: Split an SGL put between the lanes by bytes", #11947), [REDACTED:Hex High Entropy String] (Michal Shalev, "UCP/DT: Register user-supplied SGL memory handles on all required MDs", #11888).

File: src/ucp/rma/flush.c:267 (ucp_ep_flush_mem_completion) and src/ucp/rma/flush.c:278-328 (ucp_ep_flush_mem_start); triggering test test/gtest/ucp/test_ucp_rma.cc:56

Suggested fix:

Don't treat this as a PR-12233 failure — re-run the BlueField ASAN job to confirm it's a master-side flake; do not raise the gtest timeout (the process was hung, output stopped entirely for 11 min).
Audit the mem/remote-flush completion accounting: make the mem.uct_comp completion path actually finish the flush request (call req->send.flushed_cb(req) / ucp_request_complete_send() once mem.uct_comp.count reaches 0), and add an assert/trace so a lost completion fails loudly instead of hanging.
Skip ucp_proto_put_offload_update_remote_flush() for zero-length fragments so an empty IOV put (iov_count == 0) doesn't create a remote-flush entry that must later be satisfied.
To reproduce locally: UCX_MAX_RMA_LANES=2 UCX_ZCOPY_THRESH=0 ./gtest --gtest_filter='dcx/test_ucp_rma.put_nonblocking_iov_zcopy/2' with UCX_LOG_LEVEL=trace_req and check whether flush ep %p completed is ever printed for the stuck request.

Related: PR #12233 (build under test, unrelated), #11923, #11947, #11888

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 55549a3b-2764-4693-88db-ae0c10f5bedd in the triage console for the audit trail.

dmitrygx added 2 commits November 4, 2026 08:09

dmitrygx requested a review from a team as a code owner November 4, 2026 08:21

dmitrygx mentioned this pull request Nov 4, 2026

UCP: Name the peer version checks of rkey, ep recovery and AM PSN #12234

Closed

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

Comment thread src/ucp/wireup/select.c Resolved

dmitrygx mentioned this pull request Nov 4, 2026

UCP/WIREUP: Skip slow BW lanes during pairwise selection #12206

Open

dmitrygx added 2 commits November 4, 2026 09:30

dmitrygx changed the title ~~UCP/WIREUP: Name the peer version checks of the lane selection~~ UCP: Name the peer version checks Nov 4, 2026

guy-ealey-morag previously approved these changes Nov 4, 2026

View reviewed changes

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

evgeny-leksikov reviewed Nov 4, 2026

View reviewed changes

dmitrygx added 2 commits November 4, 2026 11:02

dmitrygx dismissed guy-ealey-morag’s stale review via 628c4ad November 4, 2026 11:58

dmitrygx force-pushed the topic/ucx/wireup-min-dst-version branch from aadd74a to 628c4ad Compare November 4, 2026 11:58

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

dmitrygx added 2 commits November 4, 2026 13:28

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

dmitrygx added 2 commits November 4, 2026 14:14

UCP: Print the peer version with %u in the ep recovery message

1972b90

dst_version is unsigned.

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

Comment thread src/ucp/core/ucp_types.h Resolved

Comment thread src/ucp/wireup/select.c Resolved

guy-ealey-morag reviewed Nov 4, 2026

View reviewed changes

guy-ealey-morag previously approved these changes Nov 4, 2026

View reviewed changes

dmitrygx added 2 commits November 4, 2026 20:06

UCP: Prefix the peer version constants with UCP_MIN_DST_VERSION_

76759f5

The constants are members of one enum, so give them a common prefix: UCP_<FEATURE>_MIN_DST_VERSION becomes UCP_MIN_DST_VERSION_<FEATURE>, and UCP_RELEASE_LEGACY becomes UCP_MIN_DST_VERSION_LEGACY. No functional change.

UCP: Align the values of the peer version enum

370e603

The code style check aligns the values of consecutive enum members, also across the comments between them.

dmitrygx dismissed guy-ealey-morag’s stale review via 370e603 November 4, 2026 19:12

svc-nvidia-pr-review reviewed Nov 4, 2026

View reviewed changes

dmitrygx added 2 commits November 4, 2026 23:08

UCP/WIREUP: Use UCP_MIN_DST_VERSION_LEGACY in the v1 address header

21602aa

ucp_address_unpack_header() sets dst_version of a v1 address to 0 until the release version is found after the devices, as for a peer which does not pack it. Use the named constant. No functional change.

svc-nvidia-pr-review reviewed Nov 5, 2026

View reviewed changes

Comment thread src/ucp/core/ucp_types.h Resolved

svc-nvidia-pr-review reviewed Nov 5, 2026

View reviewed changes

evgeny-leksikov approved these changes Nov 5, 2026

View reviewed changes

guy-ealey-morag approved these changes Nov 6, 2026

View reviewed changes


		/* Minimal peer release version which supports the endpoint recovery */
		#define UCP_EP_RECOVERY_MIN_DST_VERSION 22

		@@ -60,6 +60,12 @@ applications. Almost every meaningful UCX feature lives here.
		- For data sent during wireup (worker addresses, capability bits, etc.),

		} ucp_object_version_t;


		/* The dst_version of a peer which does not pack its release version */

-        ucs_diag("ep: %p: recovery support requires UCX 1.%d or later, "
+        ucs_diag("ep: %p: recovery support requires UCX 1.%d or later, "
+                 "remote peer version %u is not supported",
+                 ep, UCP_EP_RECOVERY_MIN_DST_VERSION,
+                 ucp_ep_config(ep)->key.dst_version);



		/* The dst_version of a peer which does not pack its release version */
		#define UCP_VERSION_LEGACY 0

Conversation

dmitrygx commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Uh oh!

Uh oh!

dmitrygx commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 5, 2026

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Nov 5, 2026

dmitrygx commented Nov 4, 2026 •

edited

Loading