UCT: outstanding purge implementation by jeynmann · Pull Request #11668 · openucx/ucx

jeynmann · 2026-07-17T13:05:19Z

What?

Implemente uct outstanding purge api.

Why?

How?

Add param checks in uct api.
Call ext plugin in uct api.
Add gtest cases for plugin stubs.

Rebuild the transport implementation on top of the API-only failover_ext_uct branch: - align naming to the finalized "purge" API (outstanding purge callback, params and field mask), dropping the intermediate "extract"/flags shape - switch the mlx5 external plugin path from qp_query to ep_query, with RX token derivation performed by the iface query - re-add the uct_ep_outstanding_purge()/uct_ep_failover_enable() prototypes and the uct_ib_mlx5_ext_ep_outstanding_purge() dispatch wrapper - update gtests (test_ib_mlx5_ext, test_uct_outstanding_purge) accordingly

gleon99 · 2026-07-21T07:50:25Z

+        return UCS_ERR_INVALID_PARAM;
+    }
+
+    ucs_list_for_each(plugin, &uct_ib_mlx5_ext_plugins, list) {


iface_query, ep_query, and purge each select a provider independently. Can we bind this whole flow to one provider? @evgeny-leksikov wdyt?

this is an open question, see my comment #11668 (comment)

gleon99 · 2026-07-21T07:51:01Z

            .ep_get_device_ep       = (uct_ep_get_device_ep_func_t)ucs_empty_function_return_unsupported,
-            .ep_put_sgl_zcopy       = uct_ib_mlx5_ext_ep_put_sgl_zcopy
+            .ep_put_sgl_zcopy       = uct_ib_mlx5_ext_ep_put_sgl_zcopy,
+            .ep_outstanding_purge   = uct_ib_mlx5_ext_ep_outstanding_purge


How does the caller discover purge support? QUERY_TOKEN can be advertised even when ep_query or purge is missing. Should we add a capability for purge, or require the complete query/derive/purge set?

@jeynmann unresolved?

Currently, we only need 1 cap for all token related queries and functions.
I added a helper uct_ib_mlx5_ext_find_plugin(uct_iface_h iface, uint64_t cap_flags).
When token related mask is set, we will choose the plugin with QUERY_TOKEN set in iface cap only.

gleon99 · 2026-07-21T08:49:38Z

+    }
+
+    if (ext_attr.field_mask != 0) {
+        status = uct_ib_mlx5_ext_iface_query(iface, &ext_attr);


Without a plugin, requesting token lengths now fails the entire query. Other transports return success with zero lengths and no capability bit..

@jeynmann unresolved?

I changed uct_iface_base_query_v2.
Now it returns unsupport if trying to query tokens lengh/rx token.
Therefore, without a TOKEN_QUERY supported plugin, all token related query/func will be unsupported.

gleon99 · 2026-07-21T08:50:23Z

-        iface_attr->max_put_sgl_zcopy_count = max_sgl;
+    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN) {
+        ext_attr.field_mask |= UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_TX_TOKEN;
+        ext_attr.tx_token    = (void*)iface_attr->tx_token;


This casts away the const from the public TX-token input. Can we make the plugin attribute const void * instead?

@jeynmann unresolved?

# Conflicts: # src/uct/ib/mlx5/rc/rc_mlx5_iface.c

gleon99 · 2026-07-27T11:15:54Z

@jeynmann as discussed, please make the tests much shorter and simpler, as they seem to be an overkill at the moment.

evgeny-leksikov · 2026-07-27T13:29:39Z

+#define UCT_IFACE_FLAG_V2_QUERY_TOKEN         UCS_BIT(1)  /**< Interface supports TX/RX token
+                                                               length query, RX token derivation
+                                                               from a peer TX token, endpoint TX
+                                                               token query, and outstanding
+                                                               operation purge. */


maybe just to reference to APIs that must be supported if this bit is present?

evgeny-leksikov · 2026-07-27T13:36:33Z

+ * The endpoint must be quiesced by the caller: no new operations may be posted
+ * on @a ep while this function is running, and ownership of the endpoint's


what does it mean? any UCT API is not thread safe by default

evgeny-leksikov · 2026-07-27T13:38:41Z

+ * On success, callbacks are invoked in the original endpoint posting order for
+ * operations classified as undelivered. A callback may replay the operation
+ * immediately, including before this function returns.


it looks like a certain use-case rather then description of what this function is doing

evgeny-leksikov · 2026-07-27T13:46:32Z

+ *
+ * If this function returns an error, callbacks might already have been invoked
+ * for some operations. Those callbacks remain owned by the caller; the provider
+ * reports failure for the first operation whose state could not be classified or


no need to mention any implementation specifics, e.g. provider's behavior

evgeny-leksikov · 2026-07-27T14:04:51Z

    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_CAP_FLAGS) {
-        iface_attr->cap.flags = UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY;
+        ext_attr.field_mask |= UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_CAP_FLAGS;
+    }
+
+    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN_LENGTH) {
+        ext_attr.field_mask |=
+                UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_TX_TOKEN_LEN;
+    }
+
+    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_RX_TOKEN_LENGTH) {
+        ext_attr.field_mask |=
+                UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_RX_TOKEN_LEN;
    }

-    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_MAX_PUT_SGL_ZCOPY_COUNT) {
-        iface_attr->max_put_sgl_zcopy_count = max_sgl;
+    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN) {
+        ext_attr.field_mask |= UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_TX_TOKEN;
+        ext_attr.tx_token    = iface_attr->tx_token;
+    }
+
+    if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_RX_TOKEN) {
+        ext_attr.field_mask |= UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_RX_TOKEN;
+        ext_attr.rx_token    = iface_attr->rx_token;
+    }


filling of ext_attrcould be a separate function

evgeny-leksikov · 2026-07-27T14:08:02Z

+    uct_ib_mlx5_ext_iface_query_attr_t attr = {
+        .field_mask = UCT_IB_MLX5_EXT_IFACE_QUERY_ATTR_FIELD_CAP_FLAGS
+    };


attr is "in/out-parameter", need to re-initialize on each call

evgeny-leksikov · 2026-07-27T14:10:46Z

        if (ucs_unlikely(uct_ib_mlx5_ext_is_unsupported_op(
                    (const void*)plugin->ops.iface_query))) {
            continue;
        }


do we still need this check if we have the capability?

Removing this check means we are making a strong assumption here: every plugin must provide iface query, and the capability flag is provided only by iface query. Is this ok?

IMO ucs_assert would be enough, like for ep_query

evgeny-leksikov · 2026-07-27T14:14:43Z

+        plugin = uct_ib_mlx5_ext_find_plugin(ep->iface,
+                                             UCT_IFACE_FLAG_V2_QUERY_TOKEN);
+        if ((plugin == NULL) || (uct_ib_mlx5_ext_is_unsupported_op(
+                                        (const void*)plugin->ops.ep_query))) {


according to documentation UCT_IFACE_FLAG_V2_QUERY_TOKEN must guarantee ep_query with TX token, right?

Yes. Just in case the open source plugin doesn't follow the convention. Removed.

evgeny-leksikov · 2026-07-27T14:19:31Z

+    plugin = uct_ib_mlx5_ext_find_plugin(ep->iface,
+                                         UCT_IFACE_FLAG_V2_QUERY_TOKEN);
+    if ((plugin == NULL) ||
+        (uct_ib_mlx5_ext_is_unsupported_op(
+                (const void*)plugin->ops.ep_outstanding_purge))) {
+        return UCS_ERR_UNSUPPORTED;
+    }
+
+    status = plugin->ops.ep_outstanding_purge(ep, params);


we could safely call plugin->ops.ep_outstanding_purge(ep, params); on all plugins if token contains a signature and/or versioning, then each plugin could purge its ops or return error if signature does not match

evgeny-leksikov · 2026-07-30T14:29:43Z

+    uct_rc_mlx5_iface_common_t *mlx5_iface;
+    uct_ib_mlx5_ext_iface_query_attr_t ext_attr = {};


code style: initialized variables go first

Suggested change

uct_rc_mlx5_iface_common_t *mlx5_iface;

uct_ib_mlx5_ext_iface_query_attr_t ext_attr = {};

uct_ib_mlx5_ext_iface_query_attr_t ext_attr = { 0 };

uct_rc_mlx5_iface_common_t *mlx5_iface;

evgeny-leksikov · 2026-07-30T14:33:15Z

+    if (ext_attr.field_mask != 0) {
+        status = uct_ib_mlx5_ext_iface_query(tl_iface, &ext_attr);
+        if (status != UCS_OK) {
+            return status;
+        }
+
+        if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_CAP_FLAGS) {
+            iface_attr->cap.flags |= ext_attr.cap.flags;
+        }
+
+        if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN_LENGTH) {
+            iface_attr->tx_token_length = ext_attr.tx_token_len;
+        }
+
+        if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_RX_TOKEN_LENGTH) {
+            iface_attr->rx_token_length = ext_attr.rx_token_len;
+        }
    }

    return UCS_OK;


Suggested change

if (ext_attr.field_mask != 0) {

status = uct_ib_mlx5_ext_iface_query(tl_iface, &ext_attr);

if (status != UCS_OK) {

return status;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_CAP_FLAGS) {

iface_attr->cap.flags |= ext_attr.cap.flags;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN_LENGTH) {

iface_attr->tx_token_length = ext_attr.tx_token_len;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_RX_TOKEN_LENGTH) {

iface_attr->rx_token_length = ext_attr.rx_token_len;

}

}

return UCS_OK;

if (ext_attr.field_mask == 0) {

return UCS_OK

}

status = uct_ib_mlx5_ext_iface_query(tl_iface, &ext_attr);

if (status != UCS_OK) {

return status;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_CAP_FLAGS) {

iface_attr->cap.flags |= ext_attr.cap.flags;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_TX_TOKEN_LENGTH) {

iface_attr->tx_token_length = ext_attr.tx_token_len;

}

if (iface_attr->field_mask & UCT_IFACE_ATTR_FIELD_RX_TOKEN_LENGTH) {

iface_attr->rx_token_length = ext_attr.rx_token_len;

}

return UCS_OK;

evgeny-leksikov · 2026-07-30T14:41:31Z

        if (ucs_unlikely(uct_ib_mlx5_ext_is_unsupported_op(
                    (const void*)plugin->ops.iface_query))) {
            continue;
        }


IMO ucs_assert would be enough, like for ep_query

gleon99 · 2026-08-03T11:02:21Z

@evgeny-leksikov plz re-review

evgeny-leksikov · 2026-08-03T15:06:23Z

 ucs_status_t
 uct_ib_mlx5_ext_ep_query(uct_ep_h ep, uct_ib_mlx5_ext_ep_query_attr_t *attr)
 {
+    const uint64_t token_mask = UCT_IB_MLX5_EXT_EP_QUERY_ATTR_FIELD_TX_TOKEN;


JeynmannZ added 3 commits July 17, 2026 16:01

UCT: cleanup

24f2678

UCT: keep plugin impl only

bbf1a4d

roiedanino reviewed Jul 19, 2026

View reviewed changes

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_iface.c Resolved

JeynmannZ added 2 commits July 20, 2026 05:42

UCT/IB/MLX5: address comments

e92b34c

UCT: cleanup and format

14f31b8

gleon99 requested review from evgeny-leksikov and gleon99 July 20, 2026 11:03

evgeny-leksikov reviewed Jul 20, 2026

View reviewed changes

Comment thread src/uct/ib/mlx5/ib_mlx5_ext.c Resolved

Comment thread src/uct/ib/mlx5/ib_mlx5_ext.c Resolved

Comment thread test/gtest/uct/ib/test_ib_mlx5_ext.cc Resolved

GTEST/UCT/IB: address comments

a08eb39

gleon99 reviewed Jul 21, 2026

View reviewed changes

JeynmannZ added 4 commits July 21, 2026 12:31

UCT/IB/MLX5: address comments

0e80e0a

UCT/IB/MLX5: address comments

776c898

UCT: address comments

586b72a

GTEST/UCT: address comments

7d20a98

gleon99 requested review from evgeny-leksikov and gleon99 July 23, 2026 11:26

JeynmannZ added 2 commits July 24, 2026 04:50

UCT/IB/MLX5: address comments

372af2b

Merge remote-tracking branch 'origin/master' into failover_ext_uct_impl

572dbd4

# Conflicts: # src/uct/ib/mlx5/rc/rc_mlx5_iface.c

evgeny-leksikov reviewed Jul 27, 2026

View reviewed changes

JeynmannZ added 5 commits July 28, 2026 06:44

UCT: address comments

1a0e2e6

GTEST: reduce gtest cases

0574b70

GTEST/UCT/IB: update gtest

a2188bb

GTEST/UCT/IB: fix coverity

fdb0541

GTEST/UCT/IB: fix build

7d36f0c

gleon99 requested a review from evgeny-leksikov July 30, 2026 11:11

evgeny-leksikov reviewed Jul 30, 2026

View reviewed changes

JeynmannZ added 2 commits July 31, 2026 07:22

UCT/IB/MLX5: address comments

3974080

GTEST/UCT/IB: fix gtest

fa25afd

gleon99 requested a review from evgeny-leksikov August 3, 2026 11:02

evgeny-leksikov reviewed Aug 3, 2026

View reviewed changes

JeynmannZ added 2 commits August 4, 2026 05:56

UCT: address comments

0980da9

UCT/IB/MLX5: fix format

e4fb91a

evgeny-leksikov approved these changes Aug 4, 2026

View reviewed changes

gleon99 merged commit 961504b into openucx:master Aug 6, 2026
161 checks passed

This was referenced Sep 2, 2026

[WIP][DNM]UCT/API: allow purge cb returning code #11882

Closed

UCT/MM,BASE: Fail cleanly when an MM MD cannot provide a memory handle #11683

Open

UCT/IB: Allow ODP with DDP for FW >= 49.1014 #11914

Merged

UCT/IB/MXL5: uct comp marker #11915

Merged

svc-ucx mentioned this pull request Sep 11, 2026

UCT/IB/MLX5: suppor put zcopy in purge outstanding #11874

Merged

svc-ucx mentioned this pull request Sep 28, 2026

UCP/RKEY: Keep configurations at stable addresses #11676

Merged

svc-ucx mentioned this pull request Oct 14, 2026

UCT/IB/MLX5: support short dm in purge outstanding #12058

Merged

		* The endpoint must be quiesced by the caller: no new operations may be posted
		* on @a ep while this function is running, and ownership of the endpoint's

		uct_rc_mlx5_iface_common_t *mlx5_iface;
		uct_ib_mlx5_ext_iface_query_attr_t ext_attr = {};

Conversation

jeynmann commented Jul 17, 2026

What?

Why?

How?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeynmann Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gleon99 commented Jul 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeynmann Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeynmann Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gleon99 commented Aug 3, 2026

Uh oh!

Uh oh!

jeynmann Jul 24, 2026 •

edited

Loading

jeynmann Jul 28, 2026 •

edited

Loading

jeynmann Jul 28, 2026 •

edited

Loading