The program crashed when using the LCI parcelport #6526

phil-skillwon · 2024-08-02T04:24:28Z

phil-skillwon
Aug 2, 2024

My two hosts are using Mellanox ConnectX-3 Pro network cards, set to ROCE v2 mode, and the functionality has been tested with IB tools and is working fine. I enabled the LCI-related configuration options when compiling the HPX source code, but my application crashes directly when executed. Here is the information:

python3 hpxrun.py ./HpxDemo_d.elf --parcelport lci -l 2
2024-08-02 09:40:26.617 - INFO - hpx demostration running...
2024-08-02 09:40:26.617 - INFO - hpx version: 1-10-0

{config}:
Core library:
HPX_AGAS_LOCAL_CACHE_SIZE=4096
HPX_HAVE_MALLOC=tcmalloc
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_PREFIX (configured)=
HPX_PREFIX=

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
HPX_WITH_APEX=OFF
HPX_WITH_ASYNC_MPI=OFF
HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
HPX_WITH_COROUTINE_COUNTERS=OFF
HPX_WITH_DISTRIBUTED_RUNTIME=ON
HPX_WITH_DYNAMIC_HPX_MAIN=ON
HPX_WITH_IO_COUNTERS=ON
HPX_WITH_IO_POOL=ON
HPX_WITH_ITTNOTIFY=OFF
HPX_WITH_LOGGING=ON
HPX_WITH_NETWORKING=ON
HPX_WITH_PAPI=OFF
HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
HPX_WITH_PARCELPORT_COUNTERS=OFF
HPX_WITH_PARCELPORT_GASNET=OFF
HPX_WITH_PARCELPORT_LCI=ON
HPX_WITH_PARCELPORT_LIBFABRIC=OFF
HPX_WITH_PARCELPORT_MPI=OFF
HPX_WITH_PARCELPORT_TCP=ON
HPX_WITH_PARCEL_COALESCING=ON
HPX_WITH_PARCEL_PROFILING=OFF
HPX_WITH_SANITIZERS=OFF
HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_STACKTRACES=ON
HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
HPX_WITH_TESTS_DEBUG_LOG=OFF
HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
HPX_WITH_THREAD_DEBUG_INFO=OFF
HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
HPX_WITH_THREAD_GUARD_PAGE=ON
HPX_WITH_THREAD_IDLE_RATES=OFF
HPX_WITH_THREAD_LOCAL_STORAGE=OFF
HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
HPX_WITH_THREAD_STACK_MMAP=ON
HPX_WITH_THREAD_STEALING_COUNTS=OFF
HPX_WITH_THREAD_TARGET_ADDRESS=OFF
HPX_WITH_TIMER_POOL=ON
HPX_WITH_TUPLE_RVALUE_SWAP=ON
HPX_WITH_VALGRIND=OFF
HPX_WITH_VERIFY_LOCKS=OFF
HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
HPX_WITH_WORK_REQUESTING_SCHEDULERS=ON

Module allocator_support:
HPX_ALLOCATOR_SUPPORT_WITH_CACHING=ON

Module command_line_handling_local:
HPX_COMMAND_LINE_HANDLING_LOCAL_WITH_JSON_CONFIGURATION_FILES=OFF

Module coroutines:
HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF
HPX_COROUTINES_WITH_THREAD_SCHEDULE_HINT_RUNS_AS_CHILD=OFF

Module datastructures:
HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON
HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.10.0 (AGAS: V3.0), Git: unknown
{boost}: V1.84.0
{build-type}: release
{date}: Aug 1 2024 10:05:34
{platform}: linux
{compiler}: GNU C++ version 9.4.0
{stdlib}: GNU libstdc++ version 20210601

{config}:
Core library:
HPX_AGAS_LOCAL_CACHE_SIZE=4096
HPX_HAVE_MALLOC=tcmalloc
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_PREFIX (configured)=
HPX_PREFIX=

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
HPX_WITH_APEX=OFF
HPX_WITH_ASYNC_MPI=OFF
HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
HPX_WITH_COROUTINE_COUNTERS=OFF
HPX_WITH_DISTRIBUTED_RUNTIME=ON
HPX_WITH_DYNAMIC_HPX_MAIN=ON
HPX_WITH_IO_COUNTERS=ON
HPX_WITH_IO_POOL=ON
HPX_WITH_ITTNOTIFY=OFF
HPX_WITH_LOGGING=ON
HPX_WITH_NETWORKING=ON
HPX_WITH_PAPI=OFF
HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
HPX_WITH_PARCELPORT_COUNTERS=OFF
HPX_WITH_PARCELPORT_GASNET=OFF
HPX_WITH_PARCELPORT_LCI=ON
HPX_WITH_PARCELPORT_LIBFABRIC=OFF
HPX_WITH_PARCELPORT_MPI=OFF
HPX_WITH_PARCELPORT_TCP=ON
HPX_WITH_PARCEL_COALESCING=ON
HPX_WITH_PARCEL_PROFILING=OFF
HPX_WITH_SANITIZERS=OFF
HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_STACKTRACES=ON
HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
HPX_WITH_TESTS_DEBUG_LOG=OFF
HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
HPX_WITH_THREAD_DEBUG_INFO=OFF
HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
HPX_WITH_THREAD_GUARD_PAGE=ON
HPX_WITH_THREAD_IDLE_RATES=OFF
HPX_WITH_THREAD_LOCAL_STORAGE=OFF
HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
HPX_WITH_THREAD_STACK_MMAP=ON
HPX_WITH_THREAD_STEALING_COUNTS=OFF
HPX_WITH_THREAD_TARGET_ADDRESS=OFF
HPX_WITH_TIMER_POOL=ON
HPX_WITH_TUPLE_RVALUE_SWAP=ON
HPX_WITH_VALGRIND=OFF
HPX_WITH_VERIFY_LOCKS=OFF
HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
HPX_WITH_WORK_REQUESTING_SCHEDULERS=ON

Module allocator_support:
HPX_ALLOCATOR_SUPPORT_WITH_CACHING=ON

Module command_line_handling_local:
HPX_COMMAND_LINE_HANDLING_LOCAL_WITH_JSON_CONFIGURATION_FILES=OFF

Module coroutines:
HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF
HPX_COROUTINES_WITH_THREAD_SCHEDULE_HINT_RUNS_AS_CHILD=OFF

Module datastructures:
HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON
HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.10.0 (AGAS: V3.0), Git: unknown
{boost}: V1.84.0
{build-type}: release
{date}: Aug 1 2024 10:05:34
{platform}: linux
{compiler}: GNU C++ version 9.4.0
{stdlib}: GNU libstdc++ version 20210601
{config}:
Core library:
HPX_AGAS_LOCAL_CACHE_SIZE=4096
HPX_HAVE_MALLOC=tcmalloc
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_PREFIX (configured)=
HPX_PREFIX=

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
HPX_WITH_APEX=OFF
HPX_WITH_ASYNC_MPI=OFF
HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
HPX_WITH_COROUTINE_COUNTERS=OFF
HPX_WITH_DISTRIBUTED_RUNTIME=ON
HPX_WITH_DYNAMIC_HPX_MAIN=ON
HPX_WITH_IO_COUNTERS=ON
HPX_WITH_IO_POOL=ON
HPX_WITH_ITTNOTIFY=OFF
HPX_WITH_LOGGING=ON
HPX_WITH_NETWORKING=ON
HPX_WITH_PAPI=OFF
HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
HPX_WITH_PARCELPORT_COUNTERS=OFF
HPX_WITH_PARCELPORT_GASNET=OFF
HPX_WITH_PARCELPORT_LCI=ON
HPX_WITH_PARCELPORT_LIBFABRIC=OFF
HPX_WITH_PARCELPORT_MPI=OFF
HPX_WITH_PARCELPORT_TCP=ON
HPX_WITH_PARCEL_COALESCING=ON
HPX_WITH_PARCEL_PROFILING=OFF
HPX_WITH_SANITIZERS=OFF
HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_STACKTRACES=ON
HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
HPX_WITH_TESTS_DEBUG_LOG=OFF
HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
HPX_WITH_THREAD_DEBUG_INFO=OFF
HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
HPX_WITH_THREAD_GUARD_PAGE=ON
HPX_WITH_THREAD_IDLE_RATES=OFF
HPX_WITH_THREAD_LOCAL_STORAGE=OFF
HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
HPX_WITH_THREAD_STACK_MMAP=ON
HPX_WITH_THREAD_STEALING_COUNTS=OFF
HPX_WITH_THREAD_TARGET_ADDRESS=OFF
HPX_WITH_TIMER_POOL=ON
HPX_WITH_TUPLE_RVALUE_SWAP=ON
HPX_WITH_VALGRIND=OFF
HPX_WITH_VERIFY_LOCKS=OFF
HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
HPX_WITH_WORK_REQUESTING_SCHEDULERS=ON

Module allocator_support:
HPX_ALLOCATOR_SUPPORT_WITH_CACHING=ON

Module command_line_handling_local:
HPX_COMMAND_LINE_HANDLING_LOCAL_WITH_JSON_CONFIGURATION_FILES=OFF

Module coroutines:
HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF
HPX_COROUTINES_WITH_THREAD_SCHEDULE_HINT_RUNS_AS_CHILD=OFF

Module datastructures:
HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON
HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.10.0 (AGAS: V3.0), Git: unknown
{boost}: V1.84.0
{build-type}: release
{date}: Aug 1 2024 10:05:34
{platform}: linux
{compiler}: GNU C++ version 9.4.0
{stdlib}: GNU libstdc++ version 20210601

{stack-trace}: {stack-trace}:
{stack-trace}: 19 frames:
0x7f767087a420 : /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f767087a420] in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f7671415f6d : hpx::parcelset::policies::lci::parcelport::create_connection(hpx::parcelset::locality const&, hpx::error_code&) [0x2d] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f767142a045 : hpx::parcelset::parcelport_implhpx::parcelset::policies::lci::parcelport::get_connection_and_send_parcels(hpx::parcelset::locality const&, bool) [0x295] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671418673 : /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1(+0x516673) [0x7f7671418673] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671454a7a : hpx::parcelset::detail::parcel_await_apply(hpx::parcelset::parcel&&, hpx::function<void (std::error_code const&, hpx::parcelset::parcel const&), false>&&, unsigned int, hpx::move_only_function<void (hpx::parcelset::parcel&&, hpx::function<void (std::error_code const&, hpx::parcelset::parcel const&), false>&&), false>) [0x3fa] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671415a0d : hpx::parcelset::policies::lci::parcelport::send_early_parcel(hpx::parcelset::locality const&, hpx::parcelset::parcel) [0x16d] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f767153c2f7 : /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1(+0x63a2f7) [0x7f767153c2f7] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671454a7a : hpx::parcelset::detail::parcel_await_apply(hpx::parcelset::parcel&&, hpx::function<void (std::error_code const&, hpx::parcelset::parcel const&), false>&&, unsigned int, hpx::move_only_function<void (hpx::parcelset::parcel&&, hpx::function<void (std::error_code const&, hpx::parcelset::parcel const&), false>&&), false>) [0x3fa] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f767153de79 : void hpx::agas::big_boot_barrier::apply<hpx::actions::direct_action<void ()(hpx::agas::registration_header const&), &hpx::agas::register_worker, void>, hpx::agas::registration_header>(unsigned int, unsigned int, hpx::parcelset::locality, hpx::actions::direct_action<void ()(hpx::agas::registration_header const&), &hpx::agas::register_worker, void>, hpx::agas::registration_header&&) [0x2b9] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f767153bfa8 : hpx::agas::big_boot_barrier::wait_hosted(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, void*, void*) [0x4e8] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671554e0f : hpx::runtime_distributed::initialize_agas() [0x19f] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f7671557ce3 : hpx::runtime_distributed::runtime_distributed(hpx::util::runtime_configuration&, int ()(hpx::runtime_mode), void ()()) [0x17f3] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f76713f026d : hpx::detail::run_or_start(hpx::function<int (hpx::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&, bool) [0x74d] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f76713f0e7c : hpx::detail::start_impl(hpx::function<int (hpx::program_options::variables_map&), false> const&, int, char**, hpx::init_params const&, char const*, char**) [0x6c] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x5591e403b7e8 : ./HpxDemo_d.elf(+0xb47e8) [0x5591e403b7e8] in ./HpxDemo_d.elf
0x5591e403b1f6 : ./HpxDemo_d.elf(+0xb41f6) [0x5591e403b1f6] in ./HpxDemo_d.elf
0x7f767034c083 : __libc_start_main [0xf3] in /lib/x86_64-linux-gnu/libc.so.6
0x5591e3fa028e : ./HpxDemo_d.elf(+0x1928e) [0x5591e3fa028e] in ./HpxDemo_d.elf
2024-08-02 09:40:26.636 - INFO - hpx demostration running...
2024-08-02 09:40:26.636 - INFO - hpx version: 1-10-0

{config}:
Core library:
HPX_AGAS_LOCAL_CACHE_SIZE=4096
HPX_HAVE_MALLOC=tcmalloc
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_PREFIX (configured)=
HPX_PREFIX=

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
HPX_WITH_APEX=OFF
HPX_WITH_ASYNC_MPI=OFF
HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
HPX_WITH_COROUTINE_COUNTERS=OFF
HPX_WITH_DISTRIBUTED_RUNTIME=ON
HPX_WITH_DYNAMIC_HPX_MAIN=ON
HPX_WITH_IO_COUNTERS=ON
HPX_WITH_IO_POOL=ON
HPX_WITH_ITTNOTIFY=OFF
HPX_WITH_LOGGING=ON
HPX_WITH_NETWORKING=ON
HPX_WITH_PAPI=OFF
HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
HPX_WITH_PARCELPORT_COUNTERS=OFF
HPX_WITH_PARCELPORT_GASNET=OFF
HPX_WITH_PARCELPORT_LCI=ON
HPX_WITH_PARCELPORT_LIBFABRIC=OFF
HPX_WITH_PARCELPORT_MPI=OFF
HPX_WITH_PARCELPORT_TCP=ON
HPX_WITH_PARCEL_COALESCING=ON
HPX_WITH_PARCEL_PROFILING=OFF
HPX_WITH_SANITIZERS=OFF
HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_STACKTRACES=ON
HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
HPX_WITH_TESTS_DEBUG_LOG=OFF
HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
HPX_WITH_THREAD_DEBUG_INFO=OFF
HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
HPX_WITH_THREAD_GUARD_PAGE=ON
HPX_WITH_THREAD_IDLE_RATES=OFF
HPX_WITH_THREAD_LOCAL_STORAGE=OFF
HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
HPX_WITH_THREAD_STACK_MMAP=ON
HPX_WITH_THREAD_STEALING_COUNTS=OFF
HPX_WITH_THREAD_TARGET_ADDRESS=OFF
HPX_WITH_TIMER_POOL=ON
HPX_WITH_TUPLE_RVALUE_SWAP=ON
HPX_WITH_VALGRIND=OFF
HPX_WITH_VERIFY_LOCKS=OFF
HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
HPX_WITH_WORK_REQUESTING_SCHEDULERS=ON

Module allocator_support:
HPX_ALLOCATOR_SUPPORT_WITH_CACHING=ON

Module command_line_handling_local:
HPX_COMMAND_LINE_HANDLING_LOCAL_WITH_JSON_CONFIGURATION_FILES=OFF

Module coroutines:
HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF
HPX_COROUTINES_WITH_THREAD_SCHEDULE_HINT_RUNS_AS_CHILD=OFF

Module datastructures:
HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON
HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.10.0 (AGAS: V3.0), Git: unknown
{boost}: V1.84.0
{build-type}: release
{date}: Aug 1 2024 10:05:34
{platform}: linux
{compiler}: GNU C++ version 9.4.0
{stdlib}: GNU libstdc++ version 20210601
{config}:
Core library:
HPX_AGAS_LOCAL_CACHE_SIZE=4096
HPX_HAVE_MALLOC=tcmalloc
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_PREFIX (configured)=
HPX_PREFIX=

HPX_FILESYSTEM_WITH_BOOST_FILESYSTEM_COMPATIBILITY=OFF
HPX_ITERATOR_SUPPORT_WITH_BOOST_ITERATOR_TRAVERSAL_TAG_COMPATIBILITY=OFF
HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF
HPX_WITH_APEX=OFF
HPX_WITH_ASYNC_MPI=OFF
HPX_WITH_ATTACH_DEBUGGER_ON_TEST_FAILURE=OFF
HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON
HPX_WITH_COROUTINE_COUNTERS=OFF
HPX_WITH_DISTRIBUTED_RUNTIME=ON
HPX_WITH_DYNAMIC_HPX_MAIN=ON
HPX_WITH_IO_COUNTERS=ON
HPX_WITH_IO_POOL=ON
HPX_WITH_ITTNOTIFY=OFF
HPX_WITH_LOGGING=ON
HPX_WITH_NETWORKING=ON
HPX_WITH_PAPI=OFF
HPX_WITH_PARALLEL_TESTS_BIND_NONE=OFF
HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF
HPX_WITH_PARCELPORT_COUNTERS=OFF
HPX_WITH_PARCELPORT_GASNET=OFF
HPX_WITH_PARCELPORT_LCI=ON
HPX_WITH_PARCELPORT_LIBFABRIC=OFF
HPX_WITH_PARCELPORT_MPI=OFF
HPX_WITH_PARCELPORT_TCP=ON
HPX_WITH_PARCEL_COALESCING=ON
HPX_WITH_PARCEL_PROFILING=OFF
HPX_WITH_SANITIZERS=OFF
HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF
HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF
HPX_WITH_STACKTRACES=ON
HPX_WITH_STACKTRACES_DEMANGLE_SYMBOLS=ON
HPX_WITH_STACKTRACES_STATIC_SYMBOLS=OFF
HPX_WITH_TESTS_DEBUG_LOG=OFF
HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF
HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF
HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON
HPX_WITH_THREAD_DEBUG_INFO=OFF
HPX_WITH_THREAD_DESCRIPTION_FULL=OFF
HPX_WITH_THREAD_GUARD_PAGE=ON
HPX_WITH_THREAD_IDLE_RATES=OFF
HPX_WITH_THREAD_LOCAL_STORAGE=OFF
HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON
HPX_WITH_THREAD_QUEUE_WAITTIME=OFF
HPX_WITH_THREAD_STACK_MMAP=ON
HPX_WITH_THREAD_STEALING_COUNTS=OFF
HPX_WITH_THREAD_TARGET_ADDRESS=OFF
HPX_WITH_TIMER_POOL=ON
HPX_WITH_TUPLE_RVALUE_SWAP=ON
HPX_WITH_VALGRIND=OFF
HPX_WITH_VERIFY_LOCKS=OFF
HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF
HPX_WITH_WORK_REQUESTING_SCHEDULERS=ON

Module allocator_support:
HPX_ALLOCATOR_SUPPORT_WITH_CACHING=ON

Module command_line_handling_local:
HPX_COMMAND_LINE_HANDLING_LOCAL_WITH_JSON_CONFIGURATION_FILES=OFF

Module coroutines:
HPX_COROUTINES_WITH_SWAP_CONTEXT_EMULATION=OFF
HPX_COROUTINES_WITH_THREAD_SCHEDULE_HINT_RUNS_AS_CHILD=OFF

Module datastructures:
HPX_DATASTRUCTURES_WITH_ADAPT_STD_TUPLE=ON
HPX_DATASTRUCTURES_WITH_ADAPT_STD_VARIANT=OFF

Module logging:
HPX_LOGGING_WITH_SEPARATE_DESTINATIONS=ON

Module serialization:
HPX_SERIALIZATION_WITH_ALLOW_CONST_TUPLE_MEMBERS=OFF
HPX_SERIALIZATION_WITH_ALLOW_RAW_POINTER_SERIALIZATION=OFF
HPX_SERIALIZATION_WITH_ALL_TYPES_ARE_BITWISE_SERIALIZABLE=OFF
HPX_SERIALIZATION_WITH_BOOST_TYPES=OFF
HPX_SERIALIZATION_WITH_SUPPORTS_ENDIANESS=OFF

Module topology:
HPX_TOPOLOGY_WITH_ADDITIONAL_HWLOC_TESTING=OFF

{version}: V1.10.0 (AGAS: V3.0), Git: unknown
{boost}: V1.84.0
{build-type}: release
{date}: Aug 1 2024 10:05:34
{platform}: linux
{compiler}: GNU C++ version 9.4.0
{stdlib}: GNU libstdc++ version 20210601

{stack-trace}: {stack-trace}: 10 frames:
0x7f2c8771f420 : /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f2c8771f420] in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f2c882d20a6 : /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1(+0x52b0a6) [0x7f2c882d20a6] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f2c882bb48b : hpx::parcelset::policies::lci::parcelport::io_service_work() [0x3b] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f2c882bf810 : /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1(+0x518810) [0x7f2c882bf810] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f2c882e3ed8 : asio::detail::scheduler::run(std::error_code&) [0x518] in /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
0x7f2c87bf3948 : hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) const [0xa8] in /usr/local/lib/Skillwon/Hpx/lib/libhpx_core.so
0x7f2c875ffdf4 : /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f2c875ffdf4] in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
0x7f2c87713609 : /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f2c87713609] in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f2c872ec353 : clone [0x43] in /lib/x86_64-linux-gnu/libc.so.6
Process 1 failed with an unexpected error code of -11 (expected 0)
Process 0 failed with an unexpected error code of -11 (expected 0)

Here is the gdb debug information:

gdb ./HpxDemo_d.elf

(gdb) set args --hpx:localities=2 --hpx:agas=192.168.199.219:7910 --hpx:hpx=192.168.199.219:7910
(gdb) run
Starting program: /path/to/bin/debug/HpxDemo_d.elf --hpx:localities=2 --hpx:agas=192.168.199.219:7910 --hpx:hpx=192.168.199.219:7910
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6978700 (LWP 237858)]
2024-08-02 10:03:50.330 - INFO - hpx demostration running...
2024-08-02 10:03:50.330 - INFO - hpx version: 1-10-0
2024-08-02 10:03:50.330 - INFO - hpx version string: 1.10.0
[New Thread 0x7ffff6777700 (LWP 237859)]
[New Thread 0x7ffff5f76700 (LWP 237860)]

Thread 3 "HpxDemo_d.elf" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6777700 (LWP 237859)]
0x00007ffff7ca20a6 in hpx::parcelset::policies::lci::sender_base::background_work(unsigned long) () from /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
(gdb) bt
#0 0x00007ffff7ca20a6 in hpx::parcelset::policies::lci::sender_base::background_work(unsigned long) () from /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
#1 0x00007ffff7c8b48b in hpx::parcelset::policies::lci::parcelport::io_service_work() () from /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
#2 0x00007ffff7c8f810 in asio::detail::completion_handler<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport>, asio::io_context::basic_executor_type<std::allocator, 0ul> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) () from /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
#3 0x00007ffff7cb3ed8 in asio::detail::scheduler::run(std::error_code&) () from /usr/local/lib/Skillwon/Hpx/lib/libhpx.so.1
#4 0x00007ffff75c3948 in hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) const () from /usr/local/lib/Skillwon/Hpx/lib/libhpx_core.so
#5 0x00007ffff6fcfdf4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff70e3609 in start_thread (arg=) at pthread_create.c:477
#7 0x00007ffff6cbc353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Am I using it incorrectly? How should I use LCI in an HPX-based application?

Answered by JiakunYan

Aug 23, 2024

@phil-skillwon Sorry for the delay. I am on a road trip right now.

Could you try this PR to see whether it works for you? For now, you need to manually select the GID index for specifying the env var LCI_IBV_GID_IDX. GID auto selection will come later.

View full answer

phil-skillwon · 2024-08-04T07:38:14Z

phil-skillwon
Aug 4, 2024
Author

gdb information(build hpx with debug):

Thread 3 "HpxDemo_d.elf" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5571700 (LWP 652998)]
0x00007ffff7737f15 in hpx::parcelset::policies::lci::sender_base::background_work (this=0x0) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/parcelport_lci/src/sender_base.cpp:35
35              auto completion_manager_p = pp_->get_tls_device().completion_manager_p;
(gdb) where
#0  0x00007ffff7737f15 in hpx::parcelset::policies::lci::sender_base::background_work (this=0x0) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/parcelport_lci/src/sender_base.cpp:35
#1  0x00007ffff77000a7 in hpx::parcelset::policies::lci::parcelport::io_service_work (this=0x555555f50000) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/parcelport_lci/src/parcelport_lci.cpp:266
#2  0x00007ffff772d396 in hpx::util::detail::invoke_mem_fun<void (), hpx::parcelset::policies::lci::parcelport>::operator()<hpx::parcelset::policies::lci::parcelport*&>(hpx::parcelset::policies::lci::parcelport*&) const (this=0x7ffff556ca80, 
    t1=@0x7ffff556cc30: 0x555555f50000) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/tag_invoke/include/hpx/functional/detail/invoke.hpp:110
#3  0x00007ffff772bddf in hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*>::operator()<>() & (this=0x7ffff556cc20)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/functional/include/hpx/functional/bind.hpp:187
#4  0x00007ffff77288a4 in asio::asio_handler_invoke<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*> > (function=...)
    at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/handler_invoke_hook.hpp:87
#5  0x00007ffff7724488 in asio_handler_invoke_helpers::invoke<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*>, hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*> > (function=..., context=...) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/handler_invoke_helpers.hpp:54
#6  0x00007ffff771f262 in asio::detail::handler_work<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*>, asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void>::complete<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*> > (this=0x7ffff556cbf6, function=..., handler=...)
    at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/handler_work.hpp:523
#7  0x00007ffff77195f7 in asio::detail::completion_handler<hpx::detail::bound<void (hpx::parcelset::policies::lci::parcelport::*)(), hpx::util::pack_c<unsigned long, 0ul>, hpx::parcelset::policies::lci::parcelport*>, asio::io_context::basic_executor_type<std::allocator<void>, 0ul> >::do_complete (owner=0x555555f18b00, base=0x555555f575c0) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/completion_handler.hpp:73
#8  0x00007ffff6778db4 in asio::detail::scheduler_operation::complete (this=0x555555f575c0, owner=0x555555f18b00, Python Exception <class 'gdb.error'> Couldn't find method std::_V2::error_category::name: 
ec=, bytes_transferred=0) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/scheduler_operation.hpp:39
#9  0x00007ffff677b53b in asio::detail::scheduler::do_run_one (this=0x555555f18b00, lock=..., this_thread=..., Python Exception <class 'gdb.error'> Couldn't find method std::_V2::error_category::name: 
ec=) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/impl/scheduler.ipp:492
#10 0x00007ffff677afa3 in asio::detail::scheduler::run (this=0x555555f18b00, Python Exception <class 'gdb.error'> Couldn't find method std::_V2::error_category::name: 
ec=) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/detail/impl/scheduler.ipp:209
#11 0x00007ffff6847a60 in asio::io_context::run (this=0x555555e88610) at /home/skillwon/Work/3rdParty/asio-1.28.1/include/asio/impl/io_context.ipp:63
#12 0x00007ffff68463f8 in hpx::util::io_service_pool::thread_run (this=0x555555f500c8, index=0, startup=0x0) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/io_service/src/io_service_pool.cpp:105
#13 0x00007ffff684cc70 in std::__invoke_impl<void, void (hpx::util::io_service_pool::*)(unsigned long, hpx::util::barrier*) const, hpx::util::io_service_pool*, unsigned long, hpx::util::barrier*> (__f=
    @0x555555f3aaa0: (void (hpx::util::io_service_pool::*)(const hpx::util::io_service_pool * const, unsigned long, hpx::util::barrier *)) 0x7ffff6846372 <hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) const>, 
    __t=@0x555555f3aa98: 0x555555f500c8) at /usr/include/c++/9/bits/invoke.h:73
#14 0x00007ffff684cb08 in std::__invoke<void (hpx::util::io_service_pool::*)(unsigned long, hpx::util::barrier*) const, hpx::util::io_service_pool*, unsigned long, hpx::util::barrier*> (__fn=
    @0x555555f3aaa0: (void (hpx::util::io_service_pool::*)(const hpx::util::io_service_pool * const, unsigned long, hpx::util::barrier *)) 0x7ffff6846372 <hpx::util::io_service_pool::thread_run(unsigned long, hpx::util::barrier*) const>)
    at /usr/include/c++/9/bits/invoke.h:95
#15 0x00007ffff684c9e7 in std::thread::_Invoker<std::tuple<void (hpx::util::io_service_pool::*)(unsigned long, hpx::util::barrier*) const, hpx::util::io_service_pool*, unsigned long, hpx::util::barrier*> >::_M_invoke<0ul, 1ul, 2ul, 3ul> (this=0x555555f3aa88)
    at /usr/include/c++/9/thread:244
#16 0x00007ffff684c966 in std::thread::_Invoker<std::tuple<void (hpx::util::io_service_pool::*)(unsigned long, hpx::util::barrier*) const, hpx::util::io_service_pool*, unsigned long, hpx::util::barrier*> >::operator() (this=0x555555f3aa88)
    at /usr/include/c++/9/thread:251
#17 0x00007ffff684c946 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (hpx::util::io_service_pool::*)(unsigned long, hpx::util::barrier*) const, hpx::util::io_service_pool*, unsigned long, hpx::util::barrier*> > >::_M_run (this=0x555555f3aa80)
    at /usr/include/c++/9/thread:195
#18 0x00007ffff5ff5df4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ffff6109609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x00007ffff5ce2353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0 replies

hkaiser · 2024-08-04T12:24:08Z

hkaiser
Aug 4, 2024
Maintainer

Tagging @JiakunYan

0 replies

JiakunYan · 2024-08-06T05:23:47Z

JiakunYan
Aug 6, 2024

@phil-skillwon Thanks for reporting the error!

The cause is that sender_p is still nullptr when io_service_work is executing.

#0  0x00007ffff7737f15 in hpx::parcelset::policies::lci::sender_base::background_work (this=0x0)

The io service pool is started here (inside do_run) while the send_p is initialized here and here (at the end of parcelport constructor).

In my understanding do_run should always be called after the constructor of the parcelport completes (@hkaiser please correct me if I am wrong), so this is a strange situation. Could you give me the output of ~~info locals~~ in gdb frame 1 so I can see whether setup was called or not?

#1  0x00007ffff77000a7 in hpx::parcelset::policies::lci::parcelport::io_service_work

Second thoughts: info locals will not help. p completion_managers, p devices, p prg_thread_p (essentially print some variables that have been initialized in the setup function) should help determine whether setup has been called or not.

Also, could you let me know whether you enabled HPX_WITH_FETCH_LCI when building HPX?

37 replies

phil-skillwon Aug 20, 2024
Author

I did not compile LCI separately; it was compiled together with HPX. This is my CMake configuration command when compiling HPX:

cmake .. 
-DCMAKE_FIND_DEBUG_MODE=ON -DCMAKE_BUILD_TYPE=Debug -DHPX_WITH_STATIC_LINKING=OFF -DBUILD_SHARED_LIBS=ON 
-DLCI_SERVER=ofi -DLCI_FORCE_SERVER=ON -DOFI_ROOT=/usr/lib/x86_64-linux-gnu/libfabric.so 
-DHPX_WITH_PARCELPORT_TCP=ON -DHPX_WITH_PARCELPORT_MPI=ON -DHPX_WITH_PARCELPORT_LCI=ON -DHPX_WITH_FETCH_LCI=ON -DFETCHCONTENT_SOURCE_DIR_LCI=/home/skillwon/Work/3rdParty/lci -DHPX_WITH_MALLOC=tcmalloc -DTcmalloc_INCLUDE_DIR=/usr/local/lib/Skillwon/Tcmalloc/include -DTcmalloc_LIBRARY=/usr/local/lib/Skillwon/Tcmalloc/lib/libtcmalloc_minimal.so -DBoost_ROOT=/home/skillwon/Work/3rdParty/boost_1_84_0 -DAsio_ROOT=/home/skillwon/Work/3rdParty/asio-1.28.1 -DHwloc_ROOT=/usr/local/lib/Skillwon/Hwloc -DCMAKE_INSTALL_PREFIX=/usr/local/lib/Skillwon/Hpx

However, I encountered a compilation error when compiling LCI:

[  7%] Built target lci-ucx
[  7%] Building C object _deps/lci-build/CMakeFiles/LCI.dir/lci/experimental/coll/coll.c.o
[  7%] Building C object _deps/lci-build/CMakeFiles/LCI.dir/lci/experimental/coll/barrier.c.o
[  7%] Building C object _deps/lci-build/CMakeFiles/LCI.dir/lci/experimental/coll/bcast.c.o
[  7%] Building C object _deps/lci-build/CMakeFiles/LCI.dir/lci/experimental/coll/allreduce.c.o
[  7%] Building C object _deps/lci-build/CMakeFiles/LCI.dir/lci/log/logger.c.o
In file included from /home/skillwon/Work/3rdParty/lci/lci/./backend/server.h:93,
                 from /home/skillwon/Work/3rdParty/lci/lci/./runtime/lcii.h:15,
                 from /home/skillwon/Work/3rdParty/lci/lci/experimental/coll/barrier.c:2:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h: In function ‘LCISD_post_puts’:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h:257:16: error: storage size of ‘iov’ isn’t known
  257 |   struct iovec iov;
      |                ^~~
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h: In function ‘LCISD_post_put’:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h:309:16: error: storage size of ‘iov’ isn’t known
  309 |   struct iovec iov;
      |                ^~~
In file included from /home/skillwon/Work/3rdParty/lci/lci/./backend/server.h:93,
                 from /home/skillwon/Work/3rdParty/lci/lci/./runtime/lcii.h:15,
                 from /home/skillwon/Work/3rdParty/lci/lci/experimental/coll/bcast.c:2:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h: In function ‘LCISD_post_puts’:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h:257:16: error: storage size of ‘iov’ isn’t known
  257 |   struct iovec iov;
      |                ^~~
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h: In function ‘LCISD_post_put’:
/home/skillwon/Work/3rdParty/lci/lci/./backend/ofi/server_ofi.h:309:16: error: storage size of ‘iov’ isn’t known
  309 |   struct iovec iov;
      |

So, I modified the LCI source code:

git diff
diff --git a/lci/backend/ofi/server_ofi.h b/lci/backend/ofi/server_ofi.h
index 96643a6..700cf07 100644
--- a/lci/backend/ofi/server_ofi.h
+++ b/lci/backend/ofi/server_ofi.h
@@ -7,7 +7,7 @@
 #include <rdma/fi_tagged.h>
 #include <stdlib.h>
 #include <string.h>
-
+#include <sys/uio.h>
 #include <rdma/fi_cm.h>
 #include <rdma/fi_errno.h>
 #include <rdma/fi_rma.h>

Eventually, I was able to compile successfully. Did you encounter this error when compiling in your development environment?

phil-skillwon Aug 20, 2024
Author

Following your suggestion, I recompiled LCI using the options -DLCI_SERVER=ofi -DLCI_FORCE_SERVER=ON -DOFI_ROOT=/usr/lib/x86_64-linux-gnu/libfabric.so, but the program still fails to run properly:

python3 hpxrun.py ./HpxDemo_d.elf --parcelport lci -l 1 -r mpi

The debug information output from the terminal:

2024-08-20 10:57:22.179 - INFO - hpx demostration running...
2024-08-20 10:57:22.179 - INFO - hpx version: 1-10-0
-1:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_initialize:64<lct:info:pmi> Use pmix as the PMI backend.
-1:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_initialize:79<lct:info:pmi> This process is initialized as 0/1.
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_MAX_ENDPOINTS to be 1024
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_PACKET_RETURN_THRESHOLD to be 1024
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_USE_ODP to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_TOUCH_LBUFFER to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_USE_DREG to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_USE_PREFETCH to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_PACKET_SIZE to be 12288
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_NUM_PKTS to be 65536
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_DEFAULT_QUEUE_LENGTH to be 131072
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_SENDS to be 1024
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_RECVS to be 4096
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_CQES to be 32766
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_ENABLE_EVENT_POLLING_THREAD to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_ENABLE_TD to be 1
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_ENABLE_PRG_NET_ENDPOINT to be 1
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_OFI_CXI_TRY_NO_HACK to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/runtime/env.c:LCII_env_init:119<lci:info:env> set LCI_BACKEND_TRY_LOCK_MODE to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_UCX_USE_TRY_LOCK to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_UCX_PROGRESS_FOCUSED to be 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/runtime/completion/cq.c:LCII_env_init_cq_type:23<lci:info:comp> Set LCI_CQ_TYPE to 0
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/./runtime/rendezvous.h:LCII_env_init_rdv_protocol:157<lci:info:rdv> Set LCI_RDV_PROTOCOL to 1
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:67<lci:info:ofi> Provider name: verbs;ofi_rxm
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:69<lci:info:ofi> MR mode hints: [FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, FI_MR_PROV_KEY]
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:71<lci:info:ofi> MR mode provided: [FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, FI_MR_PROV_KEY]
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:73<lci:info:ofi> Thread mode: FI_THREAD_SAFE
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:75<lci:info:ofi> Control progress mode: FI_PROGRESS_MANUAL
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:78<lci:info:ofi> Data progress mode: FI_PROGRESS_MANUAL
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:81<lci:info:ofi> Capacities: FI_LOCAL_COMM, FI_REMOTE_COMM, FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_MULTI_RECV
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:83<lci:info:ofi> Mode: 
0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:85<lci:info:ofi> Fi_info provided: fi_info:
    caps: [ FI_LOCAL_COMM, FI_REMOTE_COMM, FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_MULTI_RECV ]
    mode: [  ]
    addr_format: FI_SOCKADDR_IN
    src_addrlen: 16
    dest_addrlen: 0
    src_addr: fi_sockaddr_in://192.168.1.217:0
    dest_addr: (null)
    handle: (null)
    fi_tx_attr:
        caps: [ FI_SOURCE, FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE ]
        mode: [  ]
        op_flags: [  ]
        msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]
        comp_order: [  ]
        inject_size: 16320
        size: 1024
        iov_limit: 4
        rma_iov_limit: 1
    fi_rx_attr:
        caps: [ FI_SOURCE, FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE, FI_MULTI_RECV ]
        mode: [  ]
        op_flags: [  ]
        msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_STRICT ]
        comp_order: [  ]
        total_buffered_recv: 0
        size: 1024
        iov_limit: 4
    fi_ep_attr:
        type: FI_EP_RDM
        protocol: FI_PROTO_RXM
        protocol_version: 1
        max_msg_size: 8388608
        msg_prefix_size: 0
        max_order_raw_size: 8388608
        max_order_war_size: 0
        max_order_waw_size: 8388608
        mem_tag_format: 0xaaaaaaaaaaaaaaaa
        tx_ctx_cnt: 1
        rx_ctx_cnt: 1
        auth_key_size: 0
    fi_domain_attr:
        domain: 0x0
        name: rocep5s0
        threading: FI_THREAD_SAFE
        control_progress: FI_PROGRESS_MANUAL
        data_progress: FI_PROGRESS_MANUAL
        resource_mgmt: FI_RM_ENABLED
        av_type: FI_AV_UNSPEC
        mr_mode: [ FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, FI_MR_PROV_KEY ]
        mr_key_size: 4
        0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:87<lci:info:ofi> Fabric attributes: fi_fabric_attr:
    name: IB-0x18338657682652659712
    prov_name: verbs;ofi_rxm
    prov_version: 1.0
    api_version: 1.6

0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:89<lci:info:ofi> Domain attributes: fi_domain_attr:
    domain: 0x0
    name: rocep5s0
    threading: FI_THREAD_SAFE
    control_progress: FI_PROGRESS_MANUAL
    data_progress: FI_PROGRESS_MANUAL
    resource_mgmt: FI_RM_ENABLED
    av_type: FI_AV_UNSPEC
    mr_mode: [ FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, FI_MR_PROV_KEY ]
    mr_key_size: 4
    cq_data_size: 4
    cq_cnt: 65536
    ep_cnt: 32768
    tx_ctx_cnt: 1
    rx_ctx_cnt: 1
    max_ep_tx_ctx: 1
    max_ep_rx_ctx: 1
    max_ep_stx_ctx: 0
    max_ep_srx_ctx: 0
    cntr_cnt: 0
    mr_iov_limit: 1
    caps: [ FI_LOCAL_COMM, FI_REMOTE_COMM ]
    mode: [  ]
    auth_key_size: 0
    max_err_data: 0
    mr_cnt: 0

0:AsrockA320:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_server_init:91<lci:info:ofi> Endpoint attributes: fi_ep_attr:
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
    protocol_version: 1
    max_msg_size: 8388608
    msg_prefix_size: 0
    max_order_raw_size: 8388608
    max_order_war_size: 0
    max_order_waw_size: 8388608
    mem_tag_format: 0xaaaaaaaaaaaaaaaa
    tx_ctx_cnt: 1
    rx_ctx_cnt: 1
    auth_key_size: 0

0:451830:0:/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:LCISD_endpoint_init:158<lci:Assert failed: 0> err : No data available (/home/skillwon/Work/3rdParty/lci/lci/backend/ofi/server_ofi.c:158)
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[54272,1],0]
  Exit code:    255
--------------------------------------------------------------------------
Process 0 failed with an unexpected error code of 255 (expected 0) <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>

The error message is similar to the one before, where LCI was compiled without PMI backend support and eventually used libfabric. The current error is the same as the previous one:
#6526 (reply in thread)

server_ofi.c:LCISD_endpoint_init:158<lci:Assert failed: 0> err : No data available

JiakunYan Aug 21, 2024

The error message is similar to the one before, where LCI was compiled without PMI backend support and eventually used libfabric.

I don't really understand this. Which error were you referring to?

It seems that the "No data available" error happens when calling fi_ep_bind. I honestly don't understand what could go wrong there. You could play with the LCI_SERVER a little bit more (e.g. set them to smaller values). I will try finish the ibv fix soon.

JiakunYan Aug 23, 2024

@phil-skillwon Sorry for the delay. I am on a road trip right now.

Could you try this PR to see whether it works for you? For now, you need to manually select the GID index for specifying the env var LCI_IBV_GID_IDX. GID auto selection will come later.

Answer selected by phil-skillwon

phil-skillwon Aug 25, 2024
Author

@JiakunYan
I recompiled LCI and then ran the following test:

srun --exclusive --pty --mpi=pmi2 --nodelist=DellNode0 --ntasks=1 --ntasks-per-node=1 -p MyTestRxe gdb -ex=run HpxDemo_d.elf

The error "failed to modify QP state to RTR" has been resolved, but the program still crashed. Below is the GDB debug information:

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from HpxDemo_d.elf...
Starting program: /home/skillwon/Work/Test/Hpx/LciTest/1-DistrLog/bin/debug/HpxDemo_d.elf 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5434700 (LWP 4367)]
2024-08-25 09:35:45.357 - INFO - hpx demostration running...
2024-08-25 09:35:45.357 - INFO - hpx version: 1-10-0
-1:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_initialize:69<lct:info:pmi> The PMI backend pmix is not available.
-1:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_initialize:64<lct:info:pmi> Use pmi2 as the PMI backend.
-1:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_initialize:79<lct:info:pmi> This process is initialized as 0/1.
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_MAX_ENDPOINTS to be 1024
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_PACKET_RETURN_THRESHOLD to be 1024
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_USE_ODP to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_TOUCH_LBUFFER to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_USE_DREG to be 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_USE_PREFETCH to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_PACKET_SIZE to be 12288
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_NUM_PKTS to be 65536
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_DEFAULT_QUEUE_LENGTH to be 131072
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_SENDS to be 1024
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_RECVS to be 4096
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_SERVER_MAX_CQES to be 32766
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_ENABLE_EVENT_POLLING_THREAD to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_ENABLE_TD to be 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_IBV_GID_IDX to be 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_ENABLE_PRG_NET_ENDPOINT to be 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_OFI_CXI_TRY_NO_HACK to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/runtime/env.c:LCII_env_init:121<lci:info:env> set LCI_BACKEND_TRY_LOCK_MODE to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_UCX_USE_TRY_LOCK to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./sys/lciu_misc.h:LCIU_getenv_or:104<lci:info:env> set LCI_UCX_PROGRESS_FOCUSED to be 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/runtime/completion/cq.c:LCII_env_init_cq_type:23<lci:info:comp> Set LCI_CQ_TYPE to 0
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/./runtime/rendezvous.h:LCII_env_init_rdv_protocol:157<lci:info:rdv> Set LCI_RDV_PROTOCOL to 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/lcisi_ibv_detail.c:select_best_device_port:113<lci:info:ibv> rocep5s0:1 speed is 2 (1 x 2.500000).
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/lcisi_ibv_detail.c:select_best_device_port:130<lci:info:ibv> Select the best device rocep5s0:1.
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_server_init:168<lci:info:ibv> Maximum MTU: 4096; Active MTU: 1024
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_server_init:175<lci:info:ibv> Use GID index: 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:285<lci:info:ibv> ibv_alloc_td() failed (Operation not supported)
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_publish:89<lct:debug:pmi> publish LCI_KEY_0_0_0 13:0:00000000000000000000ffffc0a801d9
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:366<lci:info:ibv> Current inline data size is 236
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:101<lct:debug:pmi> enter pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:103<lct:debug:pmi> leave pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_getname:96<lct:debug:pmi> getname 0 LCI_KEY_0_0_0 13:0:00000000000000000000ffffc0a801d9
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:470<lci:info:ibv> qp2rank_mod is 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:101<lct:debug:pmi> enter pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:103<lct:debug:pmi> leave pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:285<lci:info:ibv> ibv_alloc_td() failed (Operation not supported)
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_publish:89<lct:debug:pmi> publish LCI_KEY_1_0_0 14:0:00000000000000000000ffffc0a801d9
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:366<lci:info:ibv> Current inline data size is 236
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:101<lct:debug:pmi> enter pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:103<lct:debug:pmi> leave pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_getname:96<lct:debug:pmi> getname 0 LCI_KEY_1_0_0 14:0:00000000000000000000ffffc0a801d9
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/backend/ibv/server_ibv.c:LCISD_endpoint_init:470<lci:info:ibv> qp2rank_mod is 1
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:101<lct:debug:pmi> enter pmi barrier
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lct/pmi/pmi_wrapper.cpp:LCT_pmi_barrier:103<lct:debug:pmi> leave pmi barrier
[New Thread 0x7ffff4a21700 (LWP 4380)]
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/runtime/matchtable/matchtable.c:initialize_ops:17<lci:info:mt> Use `hash queue` as the matching table backend.
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/runtime/device.c:LCI_device_init:71<lci:info:device> device 0x555555ed03c0 initialized
0:DellNode0:4351:0:/home/skillwon/Work/3rdParty/fixibv_lci/lci/lci/runtime/lci.c:LCI_initialize:95<lci:info:device> LCI_initialize is called
[New Thread 0x7ffff41fb700 (LWP 4381)]
[New Thread 0x7ffff39fa700 (LWP 4382)]
[New Thread 0x7ffff31f9700 (LWP 4383)]
[New Thread 0x7ffff29f8700 (LWP 4384)]
[New Thread 0x7ffff21f7700 (LWP 4385)]
[New Thread 0x7ffff19f6700 (LWP 4386)]
[New Thread 0x7ffff11f5700 (LWP 4387)]
[New Thread 0x7ffff09f4700 (LWP 4388)]

Thread 9 "HpxDemo_d.elf" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff19f6700 (LWP 4386)]
0x0000000000003404 in ?? ()
(gdb) bt
#0  0x0000000000003404 in ?? ()
#1  0x00007ffff75cb597 in hpx::util::one_size_heap_list::alloc (
    this=0x555555682b20 <hpx::util::reinitializable_static<hpx::components::detail::wrapper_heap_list<hpx::components::detail::fixed_wrapper_heap<hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, void> > >, hpx::components::detail::wrapper_heap_list<hpx::components::detail::fixed_wrapper_heap<hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, void> > >, 1ul>::data_>, count=1)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/components_base/src/server/one_size_heap_list.cpp:97
#2  0x00007ffff71970bc in hpx::lcos::detail::promise_base<int, int, hpx::lcos::detail::promise_data<int> >::init_shared_state (this=0x7ffff01f3a10)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/detail/promise_base.hpp:327
#3  0x00007ffff71912c2 in hpx::lcos::detail::promise_base<int, int, hpx::lcos::detail::promise_data<int> >::promise_base<hpx::util::thread_local_caching_allocator<char, std::allocator<char> > > (this=0x7ffff01f3a10, a=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/detail/promise_base.hpp:186
#4  0x00007ffff718de6d in hpx::distributed::promise<int, int>::promise<hpx::util::thread_local_caching_allocator<char, std::allocator<char> > > (this=0x7ffff01f3a10, a=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/promise.hpp:77
#5  0x00007ffff7975c53 in hpx::lcos::packaged_action<hpx::components::server::runtime_support::load_components_action, int, false>::packaged_action<hpx::util::thread_local_caching_allocator<char, std::allocator<char> > > (this=0x7ffff01f3a10, alloc=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/packaged_action.hpp:278
#6  0x00007ffff796aefa in hpx::detail::async_remote_impl<hpx::components::server::runtime_support::load_components_action> (policy=..., id=..., addr=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/detail/async_implementations.hpp:264
#7  0x00007ffff7967bad in hpx::detail::async_impl<hpx::components::server::runtime_support::load_components_action, hpx::detail::async_policy const&> (policy=..., id=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/detail/async_implementations.hpp:474
#8  0x00007ffff7965337 in hpx::detail::async_action_dispatch<hpx::components::server::runtime_support::load_components_action, hpx::detail::async_policy, void>::call<hpx::detail::async_policy const&> (id=..., launch_policy=...) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/async.hpp:71
#9  hpx::detail::async_action_dispatch<hpx::components::server::runtime_support::load_components_action, hpx::id_type, void>::call<>(hpx::id_type const&) (id=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/async.hpp:124
#10 hpx::async<hpx::components::server::runtime_support::load_components_action, hpx::id_type const&> (f=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/async.hpp:194
#11 hpx::components::stubs::runtime_support::load_components_async (gid=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/stubs/runtime_support_stubs.cpp:36
#12 0x00007ffff7965387 in hpx::components::stubs::runtime_support::load_components (gid=...)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/stubs/runtime_support_stubs.cpp:46
#13 0x00007ffff764ecd3 in hpx::detail::pre_main (mode=hpx::runtime_mode::console) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/init_runtime/src/pre_main.cpp:138
#14 0x00007ffff78fcf53 in hpx::runtime_distributed::run_helper(hpx::function<int (), false> const&, int&) (this=0x555555ee4b00, func=..., result=@0x555555ee4d50: 0)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/runtime_distributed.cpp:370
#15 0x00007ffff790e78f in hpx::util::detail::invoke_mem_fun<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::function<int (), false> const&, int&), hpx::runtime_distributed>::operator()<hpx::runtime_distributed*&, hpx::function<int (), false>&, int&>(hpx::runtime_distributed*&, hpx::function<int (), false>&, int&) const (this=0x7ffff01f3dc0, t1=@0x555586ff0470: 0x555555ee4b00) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/tag_invoke/include/hpx/functional/detail/invoke.hpp:111
#16 0x00007ffff790db3d in hpx::detail::bound<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::runtime_distributed::*)(hpx::function<int (), false> const&, int&), hpx::util::pack_c<unsigned long, 0ul, 1ul, 2ul>, hpx::runtime_distributed*, hpx::function<int (), false>, int&>::operator()<hpx::threads::thread_restart_state>(hpx::threads::thread_restart_state&&) & (this=0x555586ff0460) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/functional/include/hpx/functional/bind.hpp:187
#17 0x00007ffff790dba1 in hpx::threads::detail::thread_function<hpx::detail::bound<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::runtime_distributed::*)(hpx::function<int (), false> const&, int&), hpx::util::pack_c<unsigned long, 0ul, 1ul, 2ul>, hpx::runtime_distributed*, hpx::function<int (), false>, int&> >::operator()(hpx::threads::thread_restart_state) (this=0x555586ff0460)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/threading_base/include/hpx/threading_base/register_thread.hpp:35
--Type <RET> for more, q to quit, c to continue without paging--c
#18 0x00007ffff790ca13 in hpx::util::detail::callable_vtable<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::threads::thread_restart_state)>::_invoke<hpx::threads::detail::thread_function<hpx::detail::bound<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::runtime_distributed::*)(hpx::function<int (), false> const&, int&), hpx::util::pack_c<unsigned long, 0ul, 1ul, 2ul>, hpx::runtime_distributed*, hpx::function<int (), false>, int&> > >(void*, hpx::threads::thread_restart_state&&) (f=0x555586ff0460, vs#0=@0x7ffff01f3ebb: hpx::threads::thread_restart_state::signaled) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/functional/include/hpx/functional/detail/vtable/callable_vtable.hpp:88
#19 0x00007ffff66adbf0 in hpx::util::detail::basic_function<std::pair<hpx::threads::thread_schedule_state, hpx::threads::thread_id> (hpx::threads::thread_restart_state), false, false>::operator()(hpx::threads::thread_restart_state) const (vs#0=hpx::threads::thread_restart_state::signaled, this=0x555587056d68) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/functional/include/hpx/functional/detail/basic_function.hpp:236
#20 hpx::threads::coroutines::detail::coroutine_impl::operator() (this=0x555587056c50) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/coroutines/src/detail/coroutine_impl.cpp:81
#21 0x00007ffff66ac56b in hpx::threads::coroutines::detail::lx::trampoline<hpx::threads::coroutines::detail::coroutine_impl> (fun=0x555587056c50) at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/coroutines/include/hpx/coroutines/detail/context_linux_x86.hpp:181
#22 0x0000000000000000 in ?? ()
(gdb) info locals
No symbol table info available.
(gdb) frame 1
#1  0x00007ffff75cb597 in hpx::util::one_size_heap_list::alloc (
    this=0x555555682b20 <hpx::util::reinitializable_static<hpx::components::detail::wrapper_heap_list<hpx::components::detail::fixed_wrapper_heap<hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, void> > >, hpx::components::detail::wrapper_heap_list<hpx::components::detail::fixed_wrapper_heap<hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, void> > >, 1ul>::data_>, count=1)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/components_base/src/server/one_size_heap_list.cpp:97
97	        heap = create_heap_(class_name_.c_str(), heap_count_ + 1, parameters_);
(gdb) info locals
p = 0x0
result = false
heap = std::shared_ptr<hpx::util::wrapper_heap_base> (empty) = {get() = 0x0}
(gdb) frame 2
#2  0x00007ffff71970bc in hpx::lcos::detail::promise_base<int, int, hpx::lcos::detail::promise_data<int> >::init_shared_state (this=0x7ffff01f3a10)
    at /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/async_distributed/include/hpx/async_distributed/detail/promise_base.hpp:327
327	            auto ptr = hpx::components::component_heap<wrapping_type>().alloc();
(gdb) info locals
ptr = 0x7ffff01f3910
lco_ptr = std::unique_ptr<hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, void>> = {get() = 0x7ffff01f3906}
(gdb)

JiakunYan Aug 25, 2024

@phil-skillwon Thanks for testing the PR.

I am not sure this error is related to the parcelport. Since you only used one process, no communication would even go through the parcelport. @hkaiser could you take a look?

phil-skillwon Aug 25, 2024
Author

Sorry, my previous description wasn't clear enough. In fact, I also tested with two nodes, and the behavior and debug information were the same as with a single node.

srun --exclusive --pty --mpi=pmi2 --nodelist=DellNode0, AsusNode1 --ntasks=2 --ntasks-per-node=1 -p MyTestRxe gdb -ex=run HpxDemo_d.elf

JiakunYan Aug 27, 2024

@phil-skillwon Yeah, the fact that this error appears when running with a single process has already shown that this error is not related to communication, but another HPX module or your application. It looks like @hkaiser no longer monitoring this thread. I would suggest you open an issue with the error message and your application code and someone else would help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The program crashed when using the LCI parcelport #6526

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 37 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

The program crashed when using the LCI parcelport #6526

phil-skillwon Aug 2, 2024

Replies: 3 comments · 37 replies

phil-skillwon Aug 4, 2024 Author

hkaiser Aug 4, 2024 Maintainer

JiakunYan Aug 6, 2024

phil-skillwon Aug 20, 2024 Author

phil-skillwon Aug 20, 2024 Author

JiakunYan Aug 21, 2024

JiakunYan Aug 23, 2024

phil-skillwon Aug 25, 2024 Author

JiakunYan Aug 25, 2024

phil-skillwon Aug 25, 2024 Author

JiakunYan Aug 27, 2024

phil-skillwon
Aug 2, 2024

Replies: 3 comments 37 replies

phil-skillwon
Aug 4, 2024
Author

hkaiser
Aug 4, 2024
Maintainer

JiakunYan
Aug 6, 2024

phil-skillwon Aug 20, 2024
Author

phil-skillwon Aug 20, 2024
Author

phil-skillwon Aug 25, 2024
Author

phil-skillwon Aug 25, 2024
Author