prov/rxd: fixed incorrect memory access (after free) #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- fixed issue on access to fi_info object after it was removed Change-Id: Ic8d92108c6fe3bd1ac2a38a716d786507454e866 Signed-off-by: Oblomov, Sergey <[email protected]>
hoopoepg
commented
Jun 26, 2017
| @@ -1627,14 +1631,11 @@ int rxd_endpoint(struct fid_domain *domain, struct fi_info *info, | |||
| rxd_ep->do_local_mr = (rxd_domain->mr_mode & FI_MR_LOCAL) ? 1 : 0; | |||
|
|
|||
| ret = fi_endpoint(rxd_domain->dg_domain, dg_info, &rxd_ep->dg_ep, rxd_ep); | |||
| cq_attr.size = dg_info->tx_attr->size + dg_info->rx_attr->size; | |||
| fi_freeinfo(dg_info); | |||
| fi_freeinfo(dg_info); | ||
| if (ret) | ||
| goto err2; | ||
|
|
||
| memset(&cq_attr, 0, sizeof cq_attr); | ||
| cq_attr.format = FI_CQ_FORMAT_MSG; | ||
| cq_attr.size = dg_info->tx_attr->size + dg_info->rx_attr->size; |
There was a problem hiding this comment.
here - access to freed memory
|
thanks! |
shefty
pushed a commit
that referenced
this pull request
Aug 4, 2017
=================================================================
==849267== ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff4caa7230 at pc 0x7ffdf8608687 bp 0x7fff4caa71b0 sp 0x7fff4caa71a0
READ of size 8 at 0x7fff4caa7230 thread T0
#0 0x7ffdf8608686 in fi_tostr_ libfabric-current/src/fi_tostr.c:618
#1 0x402f3a in run_test_set ofi/libfabric-current/fabtest/unit/size_left_test.c:262
#2 0x403457 in main libfabric-current/fabtest/unit/size_left_test.c:317
#3 0x7ffdf4819b14 in __libc_start_main (/usr/lib64/libc.so.6+0x21b14)
#4 0x401988 in _start (libfabric-1.4.0/ofi_inst/bin/fi_size_left_test+0x401988)
Address 0x7fff4caa7230 is located at offset 32 in frame <run_test_set> of T0's stack:
This frame has 2 object(s):
[32, 36) 'ep_type'
[96, 104) 'info'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow libfabric-current/src/fi_tostr.c:618 fi_tostr_
Shadow bytes around the buggy address:
0x10006994cdf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10006994ce40: 00 00 f1 f1 f1 f1[04]f4 f4 f4 f2 f2 f2 f2 00 f4
0x10006994ce50: f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00
0x10006994ce60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10006994ce90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap righ redzone: fb
Freed Heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
ASan internal: fe
==849267== ABORTING
Signed-off-by: Sylvain Didelot <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Dec 19, 2017
Here is the deadlock scenario: #0 0x00007fed3a439495 in pthread_spin_lock () #1 0x00007fed37ad7cfd in fastlock_acquire () #2 0x00007fed37ad80a4 in psmx2_lock () #3 0x00007fed37ad8361 in psmx2_am_trx_ctxt_handler_ext () #4 0x00007fed37b084e7 in psmx2_am_trx_ctxt_handler_0 () #5 0x00007fed373c08c5 in self_am_short_request () #6 0x00007fed3739bf83 in __psm2_am_request_short () #7 0x00007fed37ad84ee in psmx2_trx_ctxt_disconnect_peers () A lock has been held in psmx2_trx_ctxt_disconnect_peers before psm2_am_request_short is called. While making progress inside this function, the execution is redirected to the AM handler due to the arrival of an incoming disconnection request. The AM handler tries to acquire the same lock that has already been held and reaches a deadlock. Fix by avoiding calling psm2_am_request_short while holding the lock. Signed-off-by: Jianxin Xiong <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Jun 1, 2020
I'm not entirely sure if it is fixes the issue our QA is seeing
(as they get err_entry.err=-104 - a wrong negative value), but
with error injection I could easily trigger a use-after-free
with the root from this function (with err_entry.err=104, though,
so I still don't know where the wrong error sign came from).
In my error injection reproducer ofi_send_socket() fails sometimes,
which then triggers free of cm_ctx without removing the fd and
cm_ctx from polling. Next poll round will then access cm_ctx and
trigger a use-after-free.
client_send_connreq
tx_cm_data
ofi_send_socket -> fails
goto err
...
err:
free(cm_ctx)
ASAN reports
READ of size 4 at 0x6120000106c8 thread T4 (rpc_poll-0)
#0 0x7f77005e0f21 in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:482
#1 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535
#2 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48
#3 0x4926dd in fi_eq_read /home/bschubert/local/rhel7/libfabric/include/rdma/fi_eq.h:352
0x6120000106c8 is located 8 bytes inside of 280-byte region [0x6120000106c0,0x6120000107d8)
freed by thread T4 (rpc_poll-0) here:
#0 0x7f77015915e7 in __interceptor_free
#1 0x7f77005e083b in client_send_connreq prov/tcp/src/tcpx_conn_mgr.c:422
#2 0x7f77005e0f7e in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:487
#3 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535
#4 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48
previously allocated by thread T5 (rpc_conn_mgr) here:
#0 0x7f7701591b7e in __interceptor_calloc
#1 0x7f77005edb5c in tcpx_ep_connect prov/tcp/src/tcpx_ep.c:103
#2 0x478b2f in fi_connect /home/bschubert/local/rhel7/libfabric/include/rdma/fi_cm.h:98
Signed-off-by: Bernd Schubert <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Jun 24, 2020
I'm not entirely sure if it is fixes the issue our QA is seeing
(as they get err_entry.err=-104 - a wrong negative value), but
with error injection I could easily trigger a use-after-free
with the root from this function (with err_entry.err=104, though,
so I still don't know where the wrong error sign came from).
In my error injection reproducer ofi_send_socket() fails sometimes,
which then triggers free of cm_ctx without removing the fd and
cm_ctx from polling. Next poll round will then access cm_ctx and
trigger a use-after-free.
client_send_connreq
tx_cm_data
ofi_send_socket -> fails
goto err
...
err:
free(cm_ctx)
ASAN reports
READ of size 4 at 0x6120000106c8 thread T4 (rpc_poll-0)
#0 0x7f77005e0f21 in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:482
#1 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535
#2 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48
#3 0x4926dd in fi_eq_read /home/bschubert/local/rhel7/libfabric/include/rdma/fi_eq.h:352
0x6120000106c8 is located 8 bytes inside of 280-byte region [0x6120000106c0,0x6120000107d8)
freed by thread T4 (rpc_poll-0) here:
#0 0x7f77015915e7 in __interceptor_free
#1 0x7f77005e083b in client_send_connreq prov/tcp/src/tcpx_conn_mgr.c:422
#2 0x7f77005e0f7e in process_cm_ctx prov/tcp/src/tcpx_conn_mgr.c:487
#3 0x7f77005e15ef in tcpx_conn_mgr_run prov/tcp/src/tcpx_conn_mgr.c:535
#4 0x7f77005fc429 in tcpx_eq_read prov/tcp/src/tcpx_eq.c:48
previously allocated by thread T5 (rpc_conn_mgr) here:
#0 0x7f7701591b7e in __interceptor_calloc
#1 0x7f77005edb5c in tcpx_ep_connect prov/tcp/src/tcpx_ep.c:103
#2 0x478b2f in fi_connect /home/bschubert/local/rhel7/libfabric/include/rdma/fi_cm.h:98
Signed-off-by: Bernd Schubert <[email protected]>
shefty
added a commit
that referenced
this pull request
Jul 27, 2020
Problem reported by Address Sanitizer:
=================================================================
==25220==ERROR: AddressSanitizer: heap-use-after-free on address 0x6270000072e0 at pc 0x00010b926a3c bp 0x700001bd1c30 sp 0x700001bd1c28
READ of size 4 at 0x6270000072e0 thread T4
#0 0x10b926a3b in sock_conn_listener_thread (libfabric.1.dylib:x86_64+0xdca3b)
#1 0x7fff7e2d5660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660)
#2 0x7fff7e2d550c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c)
#3 0x7fff7e2d4bf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8)
0x6270000072e0 is located 480 bytes inside of 12944-byte region [0x627000007100,0x62700000a390)
freed by thread T0 here:
#0 0x10baf1a9d in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56a9d)
#1 0x10b9016bf in sock_ep_close (libfabric.1.dylib:x86_64+0xb76bf)
#2 0x10b7f4a8f in fi_close fabric.h:593
#3 0x10b7f4209 in main shared_ctx.c:649
#4 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
previously allocated by thread T0 here:
#0 0x10baf1e27 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56e27)
#1 0x10b906df4 in sock_alloc_endpoint (libfabric.1.dylib:x86_64+0xbcdf4)
#2 0x10b8f7fdb in sock_msg_ep (libfabric.1.dylib:x86_64+0xadfdb)
#3 0x10b7f7c93 in fi_endpoint fi_endpoint.h:164
#4 0x10b7f5e40 in server_connect shared_ctx.c:471
#5 0x10b7f49ba in run shared_ctx.c:573
#6 0x10b7f411b in main shared_ctx.c:647
#7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
Thread T4 created by T0 here:
#0 0x10bae999d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x4e99d)
#1 0x10b925f9b in sock_conn_start_listener_thread (libfabric.1.dylib:x86_64+0xdbf9b)
#2 0x10b8e7eb2 in sock_domain (libfabric.1.dylib:x86_64+0x9deb2)
#3 0x10b7f87d3 in fi_domain fi_domain.h:306
#4 0x10b7f5c9f in server_connect shared_ctx.c:460
#5 0x10b7f49ba in run shared_ctx.c:573
#6 0x10b7f411b in main shared_ctx.c:647
#7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
The issue shows up more frequently on OS X, which emulates epoll. However, I believe the
problem could occur on any platform.
In sock_ep_close, we remove the socket from the epoll fd, then free the endpoint.
However, if the listener thread has received an event on the socket, but has not
yet started processing it, then a race can occur. The listener thread could have
returned from ofi_epoll_wait, but suspended trying to acquire the signal_lock.
The signal_lock is acquired from sock_ep_close, where ofi_epoll_del is called, then
released. The endpoint is then freed. The listener thread can now acquire the
signal_lock, where it will attempt to access the freed endpoint data.
To avoid the race, we add a change boolean to the listener. That boolean is
only changed while holding the signal_lock. When a socket is removed from the
epollfd, we mark the listener state as 'changed'. The listener thread checks the
changed state prior to processing any events. If set, it clears the state, and
calls ofi_epoll_wait again to get a new set of events to process.
Note that this works for epoll set to level-triggered (poll semantics).
Sockets that reported events will report those same events when wait is called
a second time. Sockets which were removed from the epoll set would have their
events removed, as they are no longer being monitored.
This fix is applied both to the listener thread and cm thread.
Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Jul 27, 2020
Problem reported by Address Sanitizer:
=================================================================
==25220==ERROR: AddressSanitizer: heap-use-after-free on address 0x6270000072e0 at pc 0x00010b926a3c bp 0x700001bd1c30 sp 0x700001bd1c28
READ of size 4 at 0x6270000072e0 thread T4
#0 0x10b926a3b in sock_conn_listener_thread (libfabric.1.dylib:x86_64+0xdca3b)
#1 0x7fff7e2d5660 in _pthread_body (libsystem_pthread.dylib:x86_64+0x3660)
#2 0x7fff7e2d550c in _pthread_start (libsystem_pthread.dylib:x86_64+0x350c)
#3 0x7fff7e2d4bf8 in thread_start (libsystem_pthread.dylib:x86_64+0x2bf8)
0x6270000072e0 is located 480 bytes inside of 12944-byte region [0x627000007100,0x62700000a390)
freed by thread T0 here:
#0 0x10baf1a9d in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56a9d)
#1 0x10b9016bf in sock_ep_close (libfabric.1.dylib:x86_64+0xb76bf)
#2 0x10b7f4a8f in fi_close fabric.h:593
#3 0x10b7f4209 in main shared_ctx.c:649
#4 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
previously allocated by thread T0 here:
#0 0x10baf1e27 in wrap_calloc (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56e27)
#1 0x10b906df4 in sock_alloc_endpoint (libfabric.1.dylib:x86_64+0xbcdf4)
#2 0x10b8f7fdb in sock_msg_ep (libfabric.1.dylib:x86_64+0xadfdb)
#3 0x10b7f7c93 in fi_endpoint fi_endpoint.h:164
#4 0x10b7f5e40 in server_connect shared_ctx.c:471
#5 0x10b7f49ba in run shared_ctx.c:573
#6 0x10b7f411b in main shared_ctx.c:647
#7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
Thread T4 created by T0 here:
#0 0x10bae999d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x4e99d)
#1 0x10b925f9b in sock_conn_start_listener_thread (libfabric.1.dylib:x86_64+0xdbf9b)
#2 0x10b8e7eb2 in sock_domain (libfabric.1.dylib:x86_64+0x9deb2)
#3 0x10b7f87d3 in fi_domain fi_domain.h:306
#4 0x10b7f5c9f in server_connect shared_ctx.c:460
#5 0x10b7f49ba in run shared_ctx.c:573
#6 0x10b7f411b in main shared_ctx.c:647
#7 0x7fff7dfbd014 in start (libdyld.dylib:x86_64+0x1014)
The issue shows up more frequently on OS X, which emulates epoll. However, I believe the
problem could occur on any platform.
In sock_ep_close, we remove the socket from the epoll fd, then free the endpoint.
However, if the listener thread has received an event on the socket, but has not
yet started processing it, then a race can occur. The listener thread could have
returned from ofi_epoll_wait, but suspended trying to acquire the signal_lock.
The signal_lock is acquired from sock_ep_close, where ofi_epoll_del is called, then
released. The endpoint is then freed. The listener thread can now acquire the
signal_lock, where it will attempt to access the freed endpoint data.
To avoid the race, we add a change boolean to the listener. That boolean is
only changed while holding the signal_lock. When a socket is removed from the
epollfd, we mark the listener state as 'changed'. The listener thread checks the
changed state prior to processing any events. If set, it clears the state, and
calls ofi_epoll_wait again to get a new set of events to process.
Note that this works for epoll set to level-triggered (poll semantics).
Sockets that reported events will report those same events when wait is called
a second time. Sockets which were removed from the epoll set would have their
events removed, as they are no longer being monitored.
This fix is applied both to the listener thread and cm thread.
Signed-off-by: Sean Hefty <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Dec 18, 2020
ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff4c61e7e0 at pc 0x14f2cb7ae0b9 bp 0x7fff4c61e650 sp 0x7fff4c61ddd8
WRITE of size 17 at 0x7fff4c61e7e0 thread T0
#0 0x14f2cb7ae0b8 (/lib64/libasan.so.5+0xb40b8)
#1 0x14f2cb7aedd2 in vsscanf (/lib64/libasan.so.5+0xb4dd2)
#2 0x14f2cb7aeede in __interceptor_sscanf (/lib64/libasan.so.5+0xb4ede)
#3 0x14f2cb230766 in ofi_addr_format src/common.c:401
#4 0x14f2cb233238 in ofi_str_toaddr src/common.c:780
#5 0x14f2cb314332 in vrb_handle_ib_ud_addr prov/verbs/src/verbs_info.c:1670
#6 0x14f2cb314332 in vrb_get_match_infos prov/verbs/src/verbs_info.c:1787
#7 0x14f2cb314332 in vrb_getinfo prov/verbs/src/verbs_info.c:1841
#8 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010
#9 0x14f2cb25fcc0 in ofi_get_core_info prov/util/src/util_attr.c:298
#10 0x14f2cb269b20 in ofix_getinfo prov/util/src/util_attr.c:321
#11 0x14f2cb3e29fd in rxd_getinfo prov/rxd/src/rxd_init.c:122
#12 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010
#13 0x407150 in ft_getinfo common/shared.c:794
#14 0x414917 in ft_init_fabric common/shared.c:1042
#15 0x402f40 in run functional/bw.c:155
#16 0x402f40 in main functional/bw.c:252
#17 0x14f2ca1b28e2 in __libc_start_main (/lib64/libc.so.6+0x238e2)
#18 0x401d1d in _start (/root/libfabric/fabtests/functional/fi_bw+0x401d1d)
Address 0x7fff4c61e7e0 is located in stack of thread T0 at offset 48 in frame
#0 0x14f2cb2306f3 in ofi_addr_format src/common.c:397
This frame has 1 object(s):
[32, 48) 'fmt' <== Memory access at offset 48 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.5+0xb40b8)
Shadow bytes around the buggy address:
0x1000698bbca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1000698bbcf0: 00 00 00 00 00 00 f1 f1 f1 f1 00 00[f2]f2 f3 f3
0x1000698bbd00: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
0x1000698bbd10: f1 f1 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2
0x1000698bbd20: f2 f2 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2
0x1000698bbd30: f2 f2 00 00 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00
0x1000698bbd40: 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Fixes: 5d31276 ("common: Redo address string conversions")
Signed-off-by: Honggang Li <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Dec 18, 2020
ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff4c61e7e0 at pc 0x14f2cb7ae0b9 bp 0x7fff4c61e650 sp 0x7fff4c61ddd8
WRITE of size 17 at 0x7fff4c61e7e0 thread T0
#0 0x14f2cb7ae0b8 (/lib64/libasan.so.5+0xb40b8)
#1 0x14f2cb7aedd2 in vsscanf (/lib64/libasan.so.5+0xb4dd2)
#2 0x14f2cb7aeede in __interceptor_sscanf (/lib64/libasan.so.5+0xb4ede)
#3 0x14f2cb230766 in ofi_addr_format src/common.c:401
#4 0x14f2cb233238 in ofi_str_toaddr src/common.c:780
#5 0x14f2cb314332 in vrb_handle_ib_ud_addr prov/verbs/src/verbs_info.c:1670
#6 0x14f2cb314332 in vrb_get_match_infos prov/verbs/src/verbs_info.c:1787
#7 0x14f2cb314332 in vrb_getinfo prov/verbs/src/verbs_info.c:1841
#8 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010
#9 0x14f2cb25fcc0 in ofi_get_core_info prov/util/src/util_attr.c:298
#10 0x14f2cb269b20 in ofix_getinfo prov/util/src/util_attr.c:321
#11 0x14f2cb3e29fd in rxd_getinfo prov/rxd/src/rxd_init.c:122
#12 0x14f2cb21fc28 in fi_getinfo_ src/fabric.c:1010
#13 0x407150 in ft_getinfo common/shared.c:794
#14 0x414917 in ft_init_fabric common/shared.c:1042
#15 0x402f40 in run functional/bw.c:155
#16 0x402f40 in main functional/bw.c:252
#17 0x14f2ca1b28e2 in __libc_start_main (/lib64/libc.so.6+0x238e2)
#18 0x401d1d in _start (/root/libfabric/fabtests/functional/fi_bw+0x401d1d)
Address 0x7fff4c61e7e0 is located in stack of thread T0 at offset 48 in frame
#0 0x14f2cb2306f3 in ofi_addr_format src/common.c:397
This frame has 1 object(s):
[32, 48) 'fmt' <== Memory access at offset 48 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.5+0xb40b8)
Shadow bytes around the buggy address:
0x1000698bbca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbcd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000698bbce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1000698bbcf0: 00 00 00 00 00 00 f1 f1 f1 f1 00 00[f2]f2 f3 f3
0x1000698bbd00: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
0x1000698bbd10: f1 f1 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2
0x1000698bbd20: f2 f2 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2
0x1000698bbd30: f2 f2 00 00 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00
0x1000698bbd40: 00 00 00 06 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Fixes: 5d31276 ("common: Redo address string conversions")
Signed-off-by: Honggang Li <[email protected]>
shefty
pushed a commit
that referenced
this pull request
Feb 2, 2022
Utility providers have to call fi_getinfo again to get core providers resulting in deceptive and confusing log lines where a core provider might return FI_ENODATA for a utility provider but FI_SUCCESS for the app. Extra log levels were added that say Begin/End ofi_get_core_info to make this clearer but these debug-only (not info) logs can get lost among the hundreds of lines of output. To make it easier to distinguish between log lines with and without a core provider, specifically during fi_getinfo, add a log_prefix to the log output which clarifies that the log line was outputed as part of the layered fi_getinfo call For example, the following log line sees changes as such: libfabric:53685:1643663041:verbs:fabric:vrb_get_matching_info():1514<info> checking domain: #1 mlx5_0 libfabric:53685:1643663041:ofi_rxm:verbs:fabric:vrb_get_matching_info():1514<info> checking domain: #1 mlx5_0 Signed-off-by: aingerson <[email protected]>
shefty
added a commit
that referenced
this pull request
Jan 20, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Jan 26, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Jan 28, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Feb 2, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Feb 5, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Feb 10, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
shefty
added a commit
that referenced
this pull request
Feb 16, 2023
If a posted receive matches with a saved receive, we may need to increment the rx counter. Set the rx counter increment callback to match that of the posted receive. This fixes an assert in xnet_cntr_inc() accessing a NULL cntr_inc function pointer. Program received signal SIGABRT, Aborted. 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #0 0x0000155552d4d37f in raise () from /lib64/libc.so.6 #1 0x0000155552d37db5 in abort () from /lib64/libc.so.6 #2 0x0000155552d37c89 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 0x0000155552d45a76 in __assert_fail () from /lib64/libc.so.6 #4 0x00001555522967f9 in xnet_cntr_inc (ep=0x6e4c70, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:347 #5 0x0000155552296836 in xnet_report_cntr_success (ep=0x6e4c70, cq=0x6ca930, xfer_entry=0x6f7a30) at prov/tcp/src/xnet_cq.c:354 #6 0x000015555229970d in xnet_complete_saved (saved_entry=0x6f7a30) at prov/tcp/src/xnet_progress.c:153 #7 0x0000155552299961 in xnet_recv_saved (saved_entry=0x6f7a30, rx_entry=0x6f7840) at prov/tcp/src/xnet_progress.c:188 #8 0x00001555522946f8 in xnet_srx_tag (srx=0x6dd1c0, recv_entry=0x6f7840) at prov/tcp/src/xnet_srx.c:445 #9 0x0000155552294bb1 in xnet_srx_trecv (ep_fid=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_srx.c:558 #10 0x000015555228f60e in fi_trecv (ep=0x6dd1c0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at ./include/rdma/fi_tagged.h:91 #11 0x00001555522900a7 in xnet_rdm_trecv (ep_fid=0x6d9fe0, buf=0x6990c4, len=4, desc=0x0, src_addr=0, tag=21474836494, ignore=3458764513820540928, context=0x7ffffffeb180) at prov/tcp/src/xnet_rdm.c:212 Signed-off-by: Sean Hefty <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
removed
Signed-off-by: Oblomov, Sergey [email protected]