[dmitrygx]

What

Handle multiple flush cancel w/o completion

Why ?

uct_ep_flush() with UCT_FLUSH_FLAG_CANCEL and w/o completion will be called by a user multiple times while it returns UCS_INPROGRESS
Fixes #6048
Fixes #6027
Fixes #5999
Fixes #5844

How ?

1. RC_VERBS/RC_MLX5 - remove the assertion that check FLUS_CANCEL flag
2. DC_MLX5 - if FLUSH_CANCEL is already set, don't do any actions, just move to allocating completion and returning UCS_INPROGRESS

[dmitrygx]

@brminich @hoopoepg could you review pls?
GPU nodes were offline and testing is there now. Others passed

[yosefe]

how do we avoid modify qp to error more than once?

[dmitrygx]

how do we avoid modify qp to error more than once?

good catch, thank you!
for DC_MLX5 - it is done in the 1st commit as a part of FLUSH_CANCEL handling.
for RC_VERBS/RC_MLX5 - done in the 2nd commit.

[dmitrygx]

@Artemy-Mellanox @yosefe could you review pls?