UCP/HELLO-WORLD: added error simulations by hoopoepg · Pull Request #5901 · openucx/ucx

hoopoepg · 2020-11-13T13:23:05Z

added sender/receiver/keepalive error simulations to
ucp_hello_world application

brminich · 2020-11-16T06:24:36Z

relevant coverity errors


path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:652:
switch: Switch case value "110".
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:675:
switch_case: Reached case "110".
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:677:
break: Breaking from switch.
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:711:
switch_end: Reached end of switch.
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:712:
loop: Jumping back to the beginning of the loop.
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:651:
loop_begin: Jumped back to beginning of loop.
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:651:
cond_true: Condition "(c = getopt(argc, argv, "wfbe:n:p:s:m:h")) != -1", taking true branch.
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:652:
switch: Switch case value "109".
path:/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:692:
switch_case: Reached case "109".
/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/ucp_hello_world.c:693:
var_deref_model: Passing null pointer "optarg" to "parse_mem_type", which dereferences it.
/scrap/azure/agent-05/AZP_WORKSPACE/2/s/examples/hello_world_util.h:167:5:
deref_parm_in_call: Function "strcmp" dereferences "opt_arg".
+ echo 'not ok 1 Coverity Detected 2 failures # https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=10849/ws/cov_build_devel_10849-20201114.13/output/errors/index.html'
+ echo Coverity report: 'https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=10849/ws/cov_build_devel_10849-20201114.13/output/errors/index.html'
+ printf '%s\t%s\n' Coverity 'https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=10849/ws/cov_build_devel_10849-20201114.13/output/errors/index.html'
+ module unload tools/cov

hoopoepg · 2020-11-16T11:54:50Z

yep, fixed

hoopoepg · 2020-11-16T20:12:49Z

bot:pipe:retest

hoopoepg · 2020-11-16T20:12:59Z

/azp run

azure-pipelines · 2020-11-16T20:13:13Z

Azure Pipelines successfully started running 1 pipeline(s).

dmitrygx · 2020-11-16T20:17:42Z

+    FAILURE_MODE_SEND,
+    FAILURE_MODE_RECV,
+    FAILURE_MODE_KEEPALIVE
+} ucp_failure_mode_t;


failure_mode_t
otherwise - it means that this is defined in UCP

hmmm, named way same as ucp_test_mode_t...
ok, renamed

brminich · 2020-11-17T07:34:31Z


+typedef enum {
+    FAILURE_MODE_NONE,
+    FAILURE_MODE_SEND,


you are killing either server or client, i'd better reflect it in the names instead.
SEND or RECV sounds confusing

failure mode say's what operation is failed: send/recv/keepalive

add comments : /* fail send operation on server /
/ fail receive operation on client */

added comments

hoopoepg · 2020-11-17T09:29:00Z

/azp run

azure-pipelines · 2020-11-17T09:29:13Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-11-18T06:47:19Z

bot:pipe:retest

hoopoepg · 2020-11-19T07:17:27Z

infra issues

hoopoepg · 2020-11-19T07:17:36Z

bot:pipe:retest

hoopoepg · 2020-11-19T07:17:41Z

/azp run

azure-pipelines · 2020-11-19T07:17:55Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-11-20T11:36:49Z

bot:pipe:retest

hoopoepg · 2020-11-23T09:12:04Z

@yosefe ok to squash?

yosefe · 2020-11-23T11:39:57Z


+typedef enum {
+    FAILURE_MODE_NONE,
+    FAILURE_MODE_SEND,


add comments : /* fail send operation on server /
/ fail receive operation on client */

yosefe · 2020-11-23T14:02:48Z

+    export UCX_KEEPALIVE_TIMEOUT=1s
+    export UCX_KEEPALIVE_NUM_EPS=10


yep, re-tabified

hoopoepg · 2020-11-24T06:21:09Z

/azp run

azure-pipelines · 2020-11-24T06:21:21Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-11-24T07:17:15Z

@hoopoepg pls squash

hoopoepg · 2020-11-24T07:19:30Z

yep, will do after tests are passed

hoopoepg · 2020-11-24T12:36:33Z

/azp run

azure-pipelines · 2020-11-24T12:36:46Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-11-24T22:51:19Z

/azp run

azure-pipelines · 2020-11-24T22:51:31Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-11-25T07:14:11Z

/azp run

azure-pipelines · 2020-11-25T07:14:23Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-11-25T09:43:25Z

/azp run

azure-pipelines · 2020-11-25T09:43:37Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-11-26T08:13:52Z

/azp run

azure-pipelines · 2020-11-26T08:14:05Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-11-27T11:42:23Z

@hoopoepg error seems relevant - assertion in keepalive from hello-world
https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=11252&view=logs&jobId=f6a00158-c162-5d05-e321-5b7f75412dc2&j=f6a00158-c162-5d05-e321-5b7f75412dc2&t=1b8c0ea8-5fdd-5b35-a853-1eec20ac7859

[swx-rdmz-ucx-gpu-01:20565:0:20565]  ucp_worker.c:2776 Assertion `!ucs_list_is_empty(&worker->all_eps)' failed
[swx-rdmz-ucx-gpu-01:26362:0:26362]  ucp_worker.c:2776 Assertion `!ucs_list_is_empty(&worker->all_eps)' failed

/scrap/azure/agent-06/AZP_WORKSPACE/2/s/contrib/../src/ucp/core/ucp_worker.c: [ ucp_worker_keepalive_remove_ep() ]
      ...
     2773     }
     2774 
     2775     ucs_assert(!ucs_list_is_empty(&worker->all_eps));
==>  2776 
     2777     if (ucs_list_is_only(&worker->all_eps, &ucp_ep_ext_gen(ep)->ep_list)) {
     2778         /* this is the last EP in worker */
     2779         worker->keepalive.iter = &worker->all_eps;

hoopoepg · 2020-11-27T12:32:47Z

something is wrong there: EP is trying to be removed from empty worker->all_eps
this assert is just handle situation when all_eps list is broken

hoopoepg · 2020-11-30T10:47:34Z

fix: #5972

yosefe · 2020-12-04T15:56:00Z

/azp run

azure-pipelines · 2020-12-04T15:56:12Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-12-05T12:05:02Z

/azp run

azure-pipelines · 2020-12-05T12:05:14Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-12-06T19:03:59Z

/azp run

azure-pipelines · 2020-12-06T19:04:12Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-12-11T10:37:56Z

/azp run

azure-pipelines · 2020-12-11T10:38:10Z

Azure Pipelines successfully started running 1 pipeline(s).

- added sender/receiver/keepalive error simulations to ucp_hello_world application - ucp_failure_mode_t is renamed to failure_mode_t - added keepalive tests

hoopoepg · 2020-12-12T12:09:50Z

bot:pipe:retest

hoopoepg · 2020-12-12T13:35:41Z

/azp run

azure-pipelines · 2020-12-12T13:35:54Z

Azure Pipelines successfully started running 1 pipeline(s).

hoopoepg · 2020-12-13T07:17:22Z

failed due tohttps://github.com//issues/5999

hoopoepg · 2020-12-13T07:17:28Z

/azp run

azure-pipelines · 2020-12-13T07:17:40Z

Azure Pipelines successfully started running 1 pipeline(s).

dmitrygx reviewed Nov 16, 2020

View reviewed changes

brminich reviewed Nov 17, 2020

View reviewed changes

yosefe reviewed Nov 23, 2020

View reviewed changes

yosefe approved these changes Nov 23, 2020

View reviewed changes

UCP/HELLO-WORLD: added error simulations

49e6122

- added sender/receiver/keepalive error simulations to ucp_hello_world application - ucp_failure_mode_t is renamed to failure_mode_t - added keepalive tests

hoopoepg force-pushed the topic/ucp-hello-world-improved-error-simul branch from e38ab8b to 49e6122 Compare December 11, 2020 19:40

yosefe merged commit 90cd6c9 into openucx:master Dec 13, 2020

it-is-a-robot mentioned this pull request Nov 30, 2021

Huawei 1230 kunpengcompute/hucx#49

Closed

		export UCX_KEEPALIVE_TIMEOUT=1s
		export UCX_KEEPALIVE_NUM_EPS=10

Conversation

hoopoepg commented Nov 13, 2020

Uh oh!

brminich commented Nov 16, 2020

Uh oh!

hoopoepg commented Nov 16, 2020

Uh oh!

hoopoepg commented Nov 16, 2020

Uh oh!

hoopoepg commented Nov 16, 2020

Uh oh!

azure-pipelines Bot commented Nov 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoopoepg Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoopoepg commented Nov 17, 2020

Uh oh!

azure-pipelines Bot commented Nov 17, 2020

Uh oh!

hoopoepg commented Nov 18, 2020

Uh oh!

hoopoepg commented Nov 19, 2020

Uh oh!

hoopoepg commented Nov 19, 2020

Uh oh!

hoopoepg commented Nov 19, 2020

Uh oh!

azure-pipelines Bot commented Nov 19, 2020

Uh oh!

hoopoepg commented Nov 20, 2020

Uh oh!

hoopoepg commented Nov 23, 2020

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoopoepg commented Nov 24, 2020

Uh oh!

azure-pipelines Bot commented Nov 24, 2020

Uh oh!

yosefe commented Nov 24, 2020

Uh oh!

hoopoepg commented Nov 24, 2020

Uh oh!

hoopoepg commented Nov 24, 2020

Uh oh!

azure-pipelines Bot commented Nov 24, 2020

Uh oh!

yosefe commented Nov 24, 2020

Uh oh!

azure-pipelines Bot commented Nov 24, 2020

Uh oh!

hoopoepg commented Nov 25, 2020

Uh oh!

azure-pipelines Bot commented Nov 25, 2020

Uh oh!

hoopoepg commented Nov 25, 2020

Uh oh!

hoopoepg Nov 17, 2020 •

edited

Loading