UCP/RNDV/ROCM: add support for staging rndv protocol for ROCm by souravzzz · Pull Request #5742 · openucx/ucx

souravzzz · 2020-09-25T19:40:38Z

What

Add support for host-staged transfers for ROCm D2D instead of GPUDirect RDMA

Why ?

Improves performance on platforms where GDR read performance is poor

	Latency (us)		BW (MBps)
Bytes	GDR	Staged	GDR	Staged
1	2.95	2.94	1.33	1.31
2	2.91	2.89	3.27	3.26
4	2.92	2.89	6.62	6.57
8	2.91	2.89	13.24	13.16
16	2.91	2.89	26.56	26.31
32	3.00	2.99	52.24	52.22
64	3.06	3.04	95.38	95.30
128	3.29	3.30	145.53	145.58
256	3.34	3.35	289.14	289.06
512	3.01	3.02	545.24	540.69
1024	3.09	3.10	1084.63	1073.03
2048	3.27	3.28	2210.85	2204.88
4096	3.74	3.75	4102.36	4104.76
8192	4.11	4.11	6552.18	6553.11
16384	6.16	6.10	9919.29	9905.94
32768	9.08	9.11	6721.42	6729.22
65536	12.85	12.83	7432.12	7432.12
131072	21.66	21.64	7607.14	7608.94
262144	38.41	44.13	7700.29	9162.53
524288	72.65	66.54	7747.65	12765.28
1048576	140.63	109.07	7771.32	16445.31
2097152	277.26	196.49	7780.72	18164.43
4194304	552.28	377.73	7781.99	20855.25
8388608	1099.02	753.94	7783.76	21853.04
16777216	2194.61	1164.82	7783.27	21320.62
33554432	4384.09	1940.44	7782.94	19942.70
67108864	8763.68	3435.65	7783.59	19450.68

How ?

Add new copy mechanism based on hsa_amd_memory_async_copy to rocm_copy transport
Add NEED_RKEY flag and registration cache support to rocm_copy md

@bureddy @yosefe

swx-jenkins3 · 2020-09-25T19:44:22Z

Can one of the admins verify this patch?

shamisp · 2020-09-25T19:54:23Z

ok to test

…

On Fri, Sep 25, 2020 at 2:44 PM swx-jenkins3 ***@***.***> wrote: Can one of the admins verify this patch? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5742 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARB5WCXO344OI2ZCRI3KMLSHTXKNANCNFSM4R2J2MBQ> .

souravzzz · 2020-09-25T19:59:18Z

Please add WIP-DNM label

souravzzz · 2020-10-08T21:16:54Z

@yosefe @bureddy Can you please explain what this error means? I am not able to view the report.
https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=9774&view=logs&j=087a5e36-b289-5cda-d2a1-de6f38127104&t=74a0144d-32ec-5074-48ed-ec52d134df76&l=31862

brminich · 2020-10-09T05:40:10Z

Is is coverity error, you can find the details by using the link you provided (copying it here just in case).
Also please note there is conflict with master branch in rndv.c file

2020-10-01T17:02:29.0490382Z + '[' 1 -gt 0 ']'
2020-10-01T17:02:29.0491243Z + cov-format-errors --dir /scrap/azure/agent-08/AZP_WORKSPACE/2/s/cov_build_devel_9774-20201001.14 --emacs-style
2020-10-01T17:02:29.0686818Z Error: FORWARD_NULL:
2020-10-01T17:02:29.0688119Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:787:
2020-10-01T17:02:29.0688696Z cond_true: Condition "mem_type != UCS_MEMORY_TYPE_HOST", taking true branch.
2020-10-01T17:02:29.0689402Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:791:
2020-10-01T17:02:29.0690140Z cond_true: Condition "mem_type_rma_lane != 255 /* (ucp_lane_index_t)-1 */", taking true branch.
2020-10-01T17:02:29.0690873Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:791:
2020-10-01T17:02:29.0691628Z cond_false: Condition "!(mem_type_rma_lane != 255 /* (ucp_lane_index_t)-1 */)", taking false branch.
2020-10-01T17:02:29.0692347Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:791:
2020-10-01T17:02:29.0692793Z if_end: End of if statement.
2020-10-01T17:02:29.0693370Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:798:
2020-10-01T17:02:29.0693824Z cond_true: Condition "rkey_p == NULL", taking true branch.
2020-10-01T17:02:29.0694443Z /scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:798:
2020-10-01T17:02:29.0694923Z var_compare_op: Comparing "rkey_p" to null implies that "rkey_p" might be null.
2020-10-01T17:02:29.0695601Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:801:
2020-10-01T17:02:29.0696205Z cond_true: Condition "status == UCS_OK", taking true branch.
2020-10-01T17:02:29.0696861Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:801:
2020-10-01T17:02:29.0697343Z cond_false: Condition "!(status == UCS_OK)", taking false branch.
2020-10-01T17:02:29.0697984Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:801:
2020-10-01T17:02:29.0698380Z if_end: End of if statement.
2020-10-01T17:02:29.0698942Z /scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/rndv/rndv.c:803:
2020-10-01T17:02:29.0699450Z var_deref_model: Passing null pointer "rkey_p" to "ucp_ep_rkey_unpack", which dereferences it.
2020-10-01T17:02:29.0715339Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:170:1:
2020-10-01T17:02:29.0716895Z cond_true: Condition "loc_id != 0", taking true branch.
2020-10-01T17:02:29.0717918Z /scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:170:1:
2020-10-01T17:02:29.0718666Z deref_parm_in_call: Function "ucp_ep_rkey_unpack_inner" dereferences "rkey_p".
2020-10-01T17:02:29.0719664Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:190:5:
2020-10-01T17:02:29.0720542Z cond_true: Condition "worker->flags & UCP_WORKER_FLAG_MT", taking true branch.
2020-10-01T17:02:29.0721439Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:190:5:
2020-10-01T17:02:29.0722382Z cond_true: Condition "worker->async.mode == UCS_ASYNC_MODE_THREAD_SPINLOCK", taking true branch.
2020-10-01T17:02:29.0723640Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:190:5:
2020-10-01T17:02:29.0724275Z if_fallthrough: Falling through to end of if statement.
2020-10-01T17:02:29.0725076Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:190:5:
2020-10-01T17:02:29.0725880Z if_end: End of if statement.
2020-10-01T17:02:29.0726855Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:199:5:
2020-10-01T17:02:29.0727616Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */", taking true branch.
2020-10-01T17:02:29.0728540Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:199:5:
2020-10-01T17:02:29.0729374Z cond_true: Condition "UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0730279Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:199:5:
2020-10-01T17:02:29.0731117Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */ && UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0732150Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:203:5:
2020-10-01T17:02:29.0732810Z cond_false: Condition "0 /* sizeof (md_map) <= 4 */", taking false branch.
2020-10-01T17:02:29.0733655Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:211:5:
2020-10-01T17:02:29.0734293Z cond_true: Condition "md_count <= 3", taking true branch.
2020-10-01T17:02:29.0735079Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:214:5:
2020-10-01T17:02:29.0735717Z if_fallthrough: Falling through to end of if statement.
2020-10-01T17:02:29.0736583Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:217:5:
2020-10-01T17:02:29.0737169Z if_end: End of if statement.
2020-10-01T17:02:29.0737961Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:218:5:
2020-10-01T17:02:29.0738631Z cond_false: Condition "rkey == NULL", taking false branch.
2020-10-01T17:02:29.0739425Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:221:5:
2020-10-01T17:02:29.0740004Z if_end: End of if statement.
2020-10-01T17:02:29.0740745Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0741378Z cond_true: Condition "remote_md_map", taking true branch.
2020-10-01T17:02:29.0742176Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0742822Z cond_true: Condition "remote_md_index < 64", taking true branch.
2020-10-01T17:02:29.0743612Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0744311Z cond_false: Condition "!((1UL << remote_md_index) & remote_md_map)", taking false branch.
2020-10-01T17:02:29.0745179Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0745740Z if_end: End of if statement.
2020-10-01T17:02:29.0746549Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0747219Z cond_true: Condition "remote_md_index <= 64", taking true branch.
2020-10-01T17:02:29.0748017Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0748706Z cond_false: Condition "!(remote_md_index <= 64)", taking false branch.
2020-10-01T17:02:29.0749555Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0750117Z if_end: End of if statement.
2020-10-01T17:02:29.0750867Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:248:9:
2020-10-01T17:02:29.0751761Z cond_true: Condition "(1UL << remote_md_index) & rkey->md_map", taking true branch.
2020-10-01T17:02:29.0752631Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0753286Z cond_true: Condition "rkey_index < md_count", taking true branch.
2020-10-01T17:02:29.0754103Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0754948Z cond_false: Condition "!(rkey_index < md_count)", taking false branch.
2020-10-01T17:02:29.0755806Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0756486Z if_end: End of if statement.
2020-10-01T17:02:29.0757433Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:257:13:
2020-10-01T17:02:29.0758090Z cond_true: Condition "status == UCS_OK", taking true branch.
2020-10-01T17:02:29.0758902Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:258:17:
2020-10-01T17:02:29.0759604Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */", taking true branch.
2020-10-01T17:02:29.0760568Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:258:17:
2020-10-01T17:02:29.0761316Z cond_true: Condition "UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0762220Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:258:17:
2020-10-01T17:02:29.0763058Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */ && UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0764083Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:261:13:
2020-10-01T17:02:29.0764721Z if_fallthrough: Falling through to end of if statement.
2020-10-01T17:02:29.0765502Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:270:13:
2020-10-01T17:02:29.0766158Z if_end: End of if statement.
2020-10-01T17:02:29.0766928Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:274:5:
2020-10-01T17:02:29.0767558Z loop: Jumping back to the beginning of the loop.
2020-10-01T17:02:29.0768360Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0768994Z loop_begin: Jumped back to beginning of loop.
2020-10-01T17:02:29.0769754Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0770521Z cond_true: Condition "(uint64_t)remote_md_map & (0xfffffffffffffffeULL << (uint64_t)remote_md_index)", taking true branch.
2020-10-01T17:02:29.0771450Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0772090Z cond_true: Condition "remote_md_index < 64", taking true branch.
2020-10-01T17:02:29.0772904Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0773600Z cond_false: Condition "!((1UL << remote_md_index) & remote_md_map)", taking false branch.
2020-10-01T17:02:29.0774454Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0775015Z if_end: End of if statement.
2020-10-01T17:02:29.0775754Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0776816Z cond_true: Condition "remote_md_index <= 64", taking true branch.
2020-10-01T17:02:29.0777892Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0778441Z cond_false: Condition "!(remote_md_index <= 64)", taking false branch.
2020-10-01T17:02:29.0779217Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0779652Z if_end: End of if statement.
2020-10-01T17:02:29.0780227Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:248:9:
2020-10-01T17:02:29.0780946Z cond_true: Condition "(1UL << remote_md_index) & rkey->md_map", taking true branch.
2020-10-01T17:02:29.0781662Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0782143Z cond_true: Condition "rkey_index < md_count", taking true branch.
2020-10-01T17:02:29.0782805Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0783303Z cond_false: Condition "!(rkey_index < md_count)", taking false branch.
2020-10-01T17:02:29.0783956Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0784653Z if_end: End of if statement.
2020-10-01T17:02:29.0785264Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:257:13:
2020-10-01T17:02:29.0785751Z cond_false: Condition "status == UCS_OK", taking false branch.
2020-10-01T17:02:29.0786691Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:261:20:
2020-10-01T17:02:29.0787127Z else_branch: Reached else branch.
2020-10-01T17:02:29.0787734Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:261:20:
2020-10-01T17:02:29.0788238Z cond_true: Condition "status == UCS_ERR_UNREACHABLE", taking true branch.
2020-10-01T17:02:29.0788893Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0789442Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */", taking true branch.
2020-10-01T17:02:29.0790170Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0790732Z cond_true: Condition "UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0791468Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0792139Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */ && UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0792976Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:266:13:
2020-10-01T17:02:29.0793450Z if_fallthrough: Falling through to end of if statement.
2020-10-01T17:02:29.0794089Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:270:13:
2020-10-01T17:02:29.0794493Z if_end: End of if statement.
2020-10-01T17:02:29.0795078Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:274:5:
2020-10-01T17:02:29.0795517Z loop: Jumping back to the beginning of the loop.
2020-10-01T17:02:29.0796303Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0796793Z loop_begin: Jumped back to beginning of loop.
2020-10-01T17:02:29.0797431Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0798049Z cond_true: Condition "(uint64_t)remote_md_map & (0xfffffffffffffffeULL << (uint64_t)remote_md_index)", taking true branch.
2020-10-01T17:02:29.0798851Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0799435Z cond_true: Condition "remote_md_index < 64", taking true branch.
2020-10-01T17:02:29.0800087Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0800635Z cond_false: Condition "!((1UL << remote_md_index) & remote_md_map)", taking false branch.
2020-10-01T17:02:29.0801332Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0801751Z if_end: End of if statement.
2020-10-01T17:02:29.0802317Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0802807Z cond_true: Condition "remote_md_index <= 64", taking true branch.
2020-10-01T17:02:29.0803463Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0803947Z cond_false: Condition "!(remote_md_index <= 64)", taking false branch.
2020-10-01T17:02:29.0804629Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0805049Z if_end: End of if statement.
2020-10-01T17:02:29.0805617Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:248:9:
2020-10-01T17:02:29.0806503Z cond_true: Condition "(1UL << remote_md_index) & rkey->md_map", taking true branch.
2020-10-01T17:02:29.0807241Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0807754Z cond_true: Condition "rkey_index < md_count", taking true branch.
2020-10-01T17:02:29.0808438Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0809210Z cond_false: Condition "!(rkey_index < md_count)", taking false branch.
2020-10-01T17:02:29.0809902Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:249:13:
2020-10-01T17:02:29.0810324Z if_end: End of if statement.
2020-10-01T17:02:29.0811053Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:257:13:
2020-10-01T17:02:29.0811547Z cond_false: Condition "status == UCS_OK", taking false branch.
2020-10-01T17:02:29.0812181Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:261:20:
2020-10-01T17:02:29.0812610Z else_branch: Reached else branch.
2020-10-01T17:02:29.0813204Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:261:20:
2020-10-01T17:02:29.0813693Z cond_true: Condition "status == UCS_ERR_UNREACHABLE", taking true branch.
2020-10-01T17:02:29.0814365Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0814917Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */", taking true branch.
2020-10-01T17:02:29.0815630Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0816243Z cond_true: Condition "UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0817008Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:263:17:
2020-10-01T17:02:29.0817667Z cond_true: Condition "1 /* UCS_LOG_LEVEL_TRACE <= UCS_LOG_LEVEL_TRACE_POLL */ && UCS_LOG_LEVEL_TRACE <= ucs_global_opts.log_component.log_level", taking true branch.
2020-10-01T17:02:29.0818507Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:266:13:
2020-10-01T17:02:29.0818984Z if_fallthrough: Falling through to end of if statement.
2020-10-01T17:02:29.0819605Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:270:13:
2020-10-01T17:02:29.0820018Z if_end: End of if statement.
2020-10-01T17:02:29.0820584Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:274:5:
2020-10-01T17:02:29.0821043Z loop: Jumping back to the beginning of the loop.
2020-10-01T17:02:29.0821661Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0822098Z loop_begin: Jumped back to beginning of loop.
2020-10-01T17:02:29.0822728Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0823322Z cond_true: Condition "(uint64_t)remote_md_map & (0xfffffffffffffffeULL << (uint64_t)remote_md_index)", taking true branch.
2020-10-01T17:02:29.0824074Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0824557Z cond_true: Condition "remote_md_index < 64", taking true branch.
2020-10-01T17:02:29.0825190Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0825724Z cond_false: Condition "!((1UL << remote_md_index) & remote_md_map)", taking false branch.
2020-10-01T17:02:29.0826492Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:244:9:
2020-10-01T17:02:29.0826905Z if_end: End of if statement.
2020-10-01T17:02:29.0827492Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0827982Z cond_true: Condition "remote_md_index <= 64", taking true branch.
2020-10-01T17:02:29.0828623Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0829116Z cond_false: Condition "!(remote_md_index <= 64)", taking false branch.
2020-10-01T17:02:29.0829783Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:245:9:
2020-10-01T17:02:29.0830183Z if_end: End of if statement.
2020-10-01T17:02:29.0830763Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:248:9:
2020-10-01T17:02:29.0831462Z cond_false: Condition "(1UL << remote_md_index) & rkey->md_map", taking false branch.
2020-10-01T17:02:29.0832175Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:271:9:
2020-10-01T17:02:29.0832743Z if_end: End of if statement.
2020-10-01T17:02:29.0833337Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:274:5:
2020-10-01T17:02:29.0833797Z loop: Jumping back to the beginning of the loop.
2020-10-01T17:02:29.0834399Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0834990Z loop_begin: Jumped back to beginning of loop.
2020-10-01T17:02:29.0835630Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0836266Z cond_true: Condition "(uint64_t)remote_md_map & (0xfffffffffffffffeULL << (uint64_t)remote_md_index)", taking true branch.
2020-10-01T17:02:29.0837057Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:236:5:
2020-10-01T17:02:29.0837550Z cond_false: Condition "remote_md_index < 64", taking false branch.
2020-10-01T17:02:29.0838194Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:274:5:
2020-10-01T17:02:29.0838614Z loop_end: Reached end of loop.
2020-10-01T17:02:29.0839205Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:279:5:
2020-10-01T17:02:29.0839664Z cond_true: Condition "rkey_index > 0", taking true branch.
2020-10-01T17:02:29.0840305Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:279:5:
2020-10-01T17:02:29.0841008Z cond_false: Condition "!(rkey_index > 0 || rkey->md_map == 0)", taking false branch.
2020-10-01T17:02:29.0841720Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:279:5:
2020-10-01T17:02:29.0842132Z if_end: End of if statement.
2020-10-01T17:02:29.0842697Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:286:5:
2020-10-01T17:02:29.0843410Z cond_true: Condition "worker->context->config.ext.proto_enable", taking true branch.
2020-10-01T17:02:29.0844115Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:294:9:
2020-10-01T17:02:29.0844585Z cond_false: Condition "status != UCS_OK", taking false branch.
2020-10-01T17:02:29.0845249Z path:/scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:296:9:
2020-10-01T17:02:29.0845653Z if_end: End of if statement.
2020-10-01T17:02:29.0846274Z /scrap/azure/agent-08/AZP_WORKSPACE/2/s/src/ucp/core/ucp_rkey.c:299:5:
2020-10-01T17:02:29.0846747Z deref_parm: Directly dereferencing parameter "rkey_p".
2020-10-01T17:02:29.0847652Z + echo 'not ok 1 Coverity Detected 1 failures # https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=9774/ws/cov_build_devel_9774-20201001.14/output/errors/index.html'
2020-10-01T17:02:29.0848768Z + echo Coverity report: 'https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=9774/ws/cov_build_devel_9774-20201001.14/output/errors/index.html'
2020-10-01T17:02:29.0849811Z Coverity report: https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=9774/ws/cov_build_devel_9774-20201001.14/output/errors/index.html

souravzzz · 2020-10-13T15:22:06Z

The failure seems unrelated
[ FAILED ] sockaddr/test_uct_cm_sockaddr.conn_to_non_exist_server_port/7, where GetParam() = sockaddr/enp129s0f0

@bureddy can you please remove WIP-DNM label and review the PR?

bureddy · 2020-10-15T19:27:05Z

bot:pipe:retest

bureddy · 2020-10-15T19:27:25Z

@souravzzz please resolve conflicts

souravzzz · 2020-10-15T20:04:18Z

Force pushed 690c1f9 to resolve conflicts
Will address reviews in separate commit

souravzzz · 2020-10-15T21:03:35Z

Will retest after #5797 is merged

souravzzz · 2020-10-17T07:41:04Z

bot:pipe:retest

bureddy · 2020-10-19T16:07:05Z

relevant error

2020-10-16T18:09:36.9428763Z [     INFO ] TEST: cuda <-> host
2020-10-16T18:09:37.0310255Z [swx-rdmz-ucx-gpu-02:523  :0:523] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
2020-10-16T18:09:37.4920560Z 
2020-10-16T18:09:37.4923179Z /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/core/ucp_rkey.c: [ ucp_rkey_destroy() ]
2020-10-16T18:09:37.4924328Z       ...
2020-10-16T18:09:37.4925059Z       371 
2020-10-16T18:09:37.4925884Z       372 void ucp_rkey_destroy(ucp_rkey_h rkey)
2020-10-16T18:09:37.4926678Z       373 {
2020-10-16T18:09:37.4927499Z ==>   374     unsigned remote_md_index, rkey_index;
2020-10-16T18:09:37.4928418Z       375     ucp_worker_h UCS_V_UNUSED worker;
2020-10-16T18:09:37.4929205Z       376 
2020-10-16T18:09:37.4930013Z       377     rkey_index = 0;
2020-10-16T18:09:37.4930688Z 
2020-10-16T18:09:37.9518848Z ==== backtrace (tid:    523) ====
2020-10-16T18:09:37.9520591Z  0 0x000000000005c098 ucs_debug_print_backtrace()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/debug/debug.c:656
2020-10-16T18:09:37.9521988Z  1 0x0000000000032885 ucp_rkey_destroy()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/core/ucp_rkey.c:374
2020-10-16T18:09:37.9523539Z  2 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion_inner()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1524
2020-10-16T18:09:37.9525017Z  3 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion_inner()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1525
2020-10-16T18:09:37.9526421Z  4 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1516
2020-10-16T18:09:37.9527796Z  5 0x0000000000005e82 uct_invoke_completion()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/base/uct_iface.h:671
2020-10-16T18:09:37.9529139Z  6 0x0000000000005e82 uct_cuda_copy_progress_event_queue()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/cuda/cuda_copy/cuda_copy_iface.c:146
2020-10-16T18:09:37.9530510Z  7 0x0000000000005e82 uct_cuda_copy_iface_progress()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/cuda/cuda_copy/cuda_copy_iface.c:164
2020-10-16T18:09:37.9531814Z  8 0x000000000003a3ca ucs_callbackq_dispatch()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/datastruct/callbackq.2020-10-16T18:09:36.9428763Z [     INFO ] TEST: cuda <-> host
2020-10-16T18:09:37.0310255Z [swx-rdmz-ucx-gpu-02:523  :0:523] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
2020-10-16T18:09:37.4920560Z 
2020-10-16T18:09:37.4923179Z /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/core/ucp_rkey.c: [ ucp_rkey_destroy() ]
2020-10-16T18:09:37.4924328Z       ...
2020-10-16T18:09:37.4925059Z       371 
2020-10-16T18:09:37.4925884Z       372 void ucp_rkey_destroy(ucp_rkey_h rkey)
2020-10-16T18:09:37.4926678Z       373 {
2020-10-16T18:09:37.4927499Z ==>   374     unsigned remote_md_index, rkey_index;
2020-10-16T18:09:37.4928418Z       375     ucp_worker_h UCS_V_UNUSED worker;
2020-10-16T18:09:37.4929205Z       376 
2020-10-16T18:09:37.4930013Z       377     rkey_index = 0;
2020-10-16T18:09:37.4930688Z 
2020-10-16T18:09:37.9518848Z ==== backtrace (tid:    523) ====
2020-10-16T18:09:37.9520591Z  0 0x000000000005c098 ucs_debug_print_backtrace()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/debug/debug.c:656
2020-10-16T18:09:37.9521988Z  1 0x0000000000032885 ucp_rkey_destroy()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/core/ucp_rkey.c:374
2020-10-16T18:09:37.9523539Z  2 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion_inner()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1524
2020-10-16T18:09:37.9525017Z  3 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion_inner()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1525
2020-10-16T18:09:37.9526421Z  4 0x0000000000055042 ucp_rndv_put_pipeline_frag_get_completion()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucp/rndv/rndv.c:1516
2020-10-16T18:09:37.9527796Z  5 0x0000000000005e82 uct_invoke_completion()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/base/uct_iface.h:671
2020-10-16T18:09:37.9529139Z  6 0x0000000000005e82 uct_cuda_copy_progress_event_queue()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/cuda/cuda_copy/cuda_copy_iface.c:146
2020-10-16T18:09:37.9530510Z  7 0x0000000000005e82 uct_cuda_copy_iface_progress()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/cuda/cuda_copy/cuda_copy_iface.c:164
2020-10-16T18:09:37.9531814Z  8 0x000000000003a3ca ucs_callbackq_dispatch()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/datastruct/callbackq.

bureddy · 2020-10-21T00:06:24Z

+    }
+
+    key->vaddr      = packed->vaddr;
+    key->dev_ptr    = packed->dev_ptr;


souravzzz · 2020-10-26T14:32:48Z

@yosefe Should I do a force push to resolve the conflicts or wait for your review?

yosefe · 2020-10-27T18:55:53Z

+        if (rkey_p != NULL && *rkey_p == NULL &&
+            md_attr->cap.flags & UCT_MD_FLAG_NEED_RKEY)
+        {
+            status = ucp_rkey_pack(mem_type_ep->worker->context, mdesc->memh,
+                                   &rkey_buffer, &rkey_size);
+            ucs_assert_always(status == UCS_OK);
+
+            status = ucp_ep_rkey_unpack(mem_type_ep, rkey_buffer, rkey_p);
+            ucs_assert_always(status == UCS_OK);
+            ucp_rkey_buffer_release(rkey_buffer);
+
+            ucs_assert_always(rkey_index != NULL);
+            memset(rkey_index, 0, UCP_MAX_LANES * sizeof(uint8_t));
+        }


what is this doing?

It's setting the freq->send.rndv_get.rkey to the rkey for the frag buffer so that transports that need rkey (e.g. rocm_copy) can do the D2H transfer. cuda_copy didn't need rkey so previously it was set to NULL.

(The frag buffers are registered with the memtype transports when they are allocated. This was the best way I found to retrieve the rkey.)

can we set the rkey when fragment buffer is created and save it on mdesc?

@yosefe I looked into this approach but not sure how feasible that is. My understanding is that the mdesc stores the list of uct mem handles for different mds and the ucp_rkey is created at runtime by calling ucp_rkey_pack and passing in the storage. We can probably have a pointer to the rkey inside ucp_mem_t struct but I'm not sure if that's a good idea because managing that memory becomes a problem. Thoughts?

we can manage the lifetime of rkey handle same way as we do for memh itself
maybe need a set of ucp_mpool_malloc/free functions for rndv frag pool, which will have rkey handle in addition to memh

we'd like to avoid rkey pack/unpack for every fragment, if possible, for performance reason

Is it okay to push this improvement in a follow up PR? a) This codepath doesn't get triggered for non-ROCm memory and b) the performance is bottlenecked by rocm copy performance which is much larger than cost of packing rkey.

@yosefe I have rebased onto latest master and squashed the changes.

hoopoepg · 2020-11-23T09:28:07Z

@souravzzz PR is conflicting with master

souravzzz · 2020-11-23T17:08:02Z

@hoopoepg Resolved conflicts and rebased on latest master

souravzzz · 2020-12-03T22:12:04Z

Updated to rebase on latest master and fix conflict with 583b37f

yosefe · 2020-12-04T14:56:04Z

Updated to rebase on latest master and fix conflict with 583b37f

@souravzzz pls avoid force push during review, use only merge and not rebase, force push doesn't allow to review only the changes since last review cycle
see https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines section 8

yosefe · 2020-12-04T15:05:09Z

RNDV common part is good now, only minor style comment on ROCM code

souravzzz · 2020-12-04T15:19:03Z

@yosefe Understood. In case of ad110e4 I had to do it because 583b37f changed ucs_ternary_t to ucs_ternary_auto_value_t and I had to pull in those changes for the build to pass. I didn't realize that it would hide the previous individual commits. Will avoid in the future.

yosefe

pls squash

yosefe · 2020-12-05T12:04:55Z

/azp run

azure-pipelines · 2020-12-05T12:05:08Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-12-06T18:03:25Z

/azp run

azure-pipelines · 2020-12-06T18:03:38Z

Azure Pipelines successfully started running 1 pipeline(s).

bureddy added the WIP-DNM Work in progress / Do not review label Sep 25, 2020

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 99ceacb to f1c80f7 Compare October 1, 2020 15:11

souravzzz force-pushed the sourav/rocm-staged-d2d branch from f1c80f7 to 6419267 Compare October 13, 2020 02:45

shamisp requested a review from bureddy October 14, 2020 18:06

shamisp removed the WIP-DNM Work in progress / Do not review label Oct 14, 2020

bureddy reviewed Oct 15, 2020

View reviewed changes

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 6419267 to 690c1f9 Compare October 15, 2020 20:02

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 690c1f9 to a877c3f Compare October 16, 2020 16:52

bureddy reviewed Oct 21, 2020

View reviewed changes

bureddy approved these changes Oct 21, 2020

View reviewed changes

bureddy requested a review from yosefe October 21, 2020 16:55

souravzzz mentioned this pull request Oct 23, 2020

UCT/ROCM/COPY: Use hipMemcpy for large message copies #4672

Closed

yosefe reviewed Oct 27, 2020

View reviewed changes

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 0c38ce0 to ea5aaa2 Compare November 3, 2020 19:51

yosefe reviewed Nov 4, 2020

View reviewed changes

Comment thread src/ucp/rndv/rndv.c Resolved

souravzzz force-pushed the sourav/rocm-staged-d2d branch from ea5aaa2 to 973003e Compare November 11, 2020 16:40

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 973003e to 0ed167e Compare November 23, 2020 15:37

hoopoepg reviewed Nov 24, 2020

View reviewed changes

Comment thread src/uct/rocm/copy/rocm_copy_ep.c Resolved

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 724732c to ad110e4 Compare December 3, 2020 22:12

yosefe reviewed Dec 4, 2020

View reviewed changes

Comment thread src/uct/rocm/copy/rocm_copy_md.c Resolved

yosefe reviewed Dec 4, 2020

View reviewed changes

Comment thread src/uct/rocm/copy/rocm_copy_ep.c Resolved

Comment thread src/uct/rocm/copy/rocm_copy_ep.c Resolved

Comment thread src/uct/rocm/copy/rocm_copy_ep.c Resolved

yosefe approved these changes Dec 4, 2020

View reviewed changes

UCP/RNDV/ROCM: add support for staging rndv protocol for ROCm

697b0a0

souravzzz force-pushed the sourav/rocm-staged-d2d branch from 00de81e to 697b0a0 Compare December 5, 2020 16:59

yosefe merged commit 497c593 into openucx:master Dec 7, 2020

souravzzz deleted the sourav/rocm-staged-d2d branch December 7, 2020 14:45

it-is-a-robot mentioned this pull request Nov 30, 2021

Huawei 1230 kunpengcompute/hucx#49

Closed

Conversation

souravzzz commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why ?

How ?

Uh oh!

swx-jenkins3 commented Sep 25, 2020

Uh oh!

shamisp commented Sep 25, 2020 via email

Uh oh!

souravzzz commented Sep 25, 2020

Uh oh!

souravzzz commented Oct 8, 2020

Uh oh!

brminich commented Oct 9, 2020

Uh oh!

souravzzz commented Oct 13, 2020

Uh oh!

bureddy commented Oct 15, 2020

Uh oh!

bureddy commented Oct 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souravzzz commented Oct 15, 2020

Uh oh!

souravzzz commented Oct 15, 2020

Uh oh!

souravzzz commented Oct 17, 2020

Uh oh!

bureddy commented Oct 19, 2020

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

souravzzz commented Oct 26, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

souravzzz Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hoopoepg commented Nov 23, 2020

Uh oh!

souravzzz commented Nov 23, 2020

Uh oh!

Uh oh!

souravzzz commented Dec 3, 2020

Uh oh!

yosefe commented Dec 4, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souravzzz commented Sep 25, 2020 •

edited

Loading

bureddy commented Oct 15, 2020 •

edited

Loading

souravzzz Oct 27, 2020 •

edited

Loading

souravzzz commented Dec 4, 2020 •

edited

Loading