searchd crashed after a few inserts from http json interface

[chiacy]

Bug Description:

searchd crashed by itself after a 1 -2 minutes and keeps restarting.

only realtime index is used, and there are only 2 indexes which is almost identical.

------- FATAL: CRASH DUMP -------
[Tue May 28 18:45:19.092 2024] [313838]

--- crashed invalid query ---

--- request dump end ---
--- local index:
Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (jammy) (cross-compiled)
Stack bottom = 0x7373f805dae0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7373f8060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x227)[0x583cca879057]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x364)[0x583cca6ef224]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x73741de42520]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN7hnswlib15HierarchicalNSWIfED2Ev+0x43)[0x73741d57fad3]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN7hnswlib15HierarchicalNSWIfED0Ev+0x9)[0x73741d57fc09]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN3knn11HNSWIndex_cD0Ev+0x35)[0x73741d57d705]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN3knn5KNN_cD0Ev+0xce)[0x73741d57d8ee]
/usr/bin/searchd(_ZN13CSphIndex_VLND1Ev+0xc5)[0x583cca77d0a5]
/usr/bin/searchd(_ZN13CSphIndex_VLND0Ev+0x9)[0x583cca77d2d9]
/usr/bin/searchd(_ZNK9RtIndex_c17PreallocDiskChunkERK10CSphStringiP17FilenameBuilder_iRN3sph8Vector_TIS0_NS5_13DefaultCopy_TIS0_EENS5_14DefaultRelimitENS5_16DefaultStorage_TIS0_EEEERS0_PKc+0x214)[0x583cca960424]
/usr/bin/searchd(_ZN9RtIndex_c13SaveDiskChunkEbbb+0x67e)[0x583cca95afde]
/usr/bin/searchd(ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB+0x1c)[0x583ccb893c1c]
/usr/bin/searchd(make_fcontext+0x37)[0x583ccb8d5b17]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x000073741DE42520 in /lib/x86_64-linux-gnu/libc.so.6
3# hnswlib::HierarchicalNSW::~HierarchicalNSW() in /usr/share/manticore/modules/lib_manticore_knn.so
4# hnswlib::HierarchicalNSW::~HierarchicalNSW() in /usr/share/manticore/modules/lib_manticore_knn.so
5# knn::HNSWIndex_c::~HNSWIndex_c() in /usr/share/manticore/modules/lib_manticore_knn.so
6# knn::KNN_c::~KNN_c() in /usr/share/manticore/modules/lib_manticore_knn.so
7# CSphIndex_VLN::~CSphIndex_VLN() in /usr/bin/searchd
8# CSphIndex_VLN::~CSphIndex_VLN() in /usr/bin/searchd
9# RtIndex_c::PreallocDiskChunk(CSphString const&, int, FilenameBuilder_i*, sph::Vector_T<CSphString, sph::DefaultCopy_T, sph::DefaultRelimit, sph::DefaultStorage_T >&, CSphString&, char const*) const in /usr/bin/searchd
10# RtIndex_c::SaveDiskChunk(bool, bool, bool) in /usr/bin/searchd
11# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
12# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
--- Totally 1 threads, and 0 client-working threads ---
------- CRASH DUMP END -------
[Tue May 28 18:45:22.177 2024] [313696] watchdog: main process 313838 crashed via CRASH_EXIT (exit code 2), will be restarted
[Tue May 28 18:45:22.177 2024] [313696] watchdog: main process 313858 forked ok
[Tue May 28 18:45:22.178 2024] [313858] Using local time zone '/etc/localtime'
[Tue May 28 18:45:22.178 2024] [313858] starting daemon version '6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
[Tue May 28 18:45:22.178 2024] [313858] listening on 127.0.0.1:9312 for sphinx and http(s)
[Tue May 28 18:45:22.178 2024] [313858] listening on all interfaces for mysql, port=9306
[Tue May 28 18:45:22.178 2024] [313858] listening on all interfaces for sphinx and http(s), port=9308
[Tue May 28 18:45:22.281 2024] [313861] prereading 2 tables
[Tue May 28 18:45:22.281 2024] [313861] preread 2 tables in 0.000 sec
[Tue May 28 18:45:22.289 2024] [313858] accepting connections
[Tue May 28 18:45:22.331 2024] [313861] [BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://127.0.0.1:9312 --bind=127.0.0.1 --threads=4 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:33495
[Tue May 28 18:45:22.331 2024] [313861] [BUDDY] Loaded plugins:
[Tue May 28 18:45:22.331 2024] [313861] [BUDDY] core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[Tue May 28 18:45:22.331 2024] [313861] [BUDDY] local:
[Tue May 28 18:45:22.331 2024] [313861] [BUDDY] extra:
------- FATAL: CRASH DUMP -------
[Tue May 28 18:45:34.302 2024] [313858]

--- crashed invalid query ---

--- request dump end ---
--- local index:
Manticore 6.3.0 1811a9efb@24052209 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (jammy) (cross-compiled)
Stack bottom = 0x7373f805dae0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7373f8060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x227)[0x583cca879057]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x364)[0x583cca6ef224]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x73741de42520]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN7hnswlib15HierarchicalNSWIfED2Ev+0x43)[0x73741d57fad3]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN7hnswlib15HierarchicalNSWIfED0Ev+0x9)[0x73741d57fc09]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN3knn11HNSWIndex_cD0Ev+0x35)[0x73741d57d705]
/usr/share/manticore/modules/lib_manticore_knn.so(_ZN3knn5KNN_cD0Ev+0xce)[0x73741d57d8ee]
/usr/bin/searchd(_ZN13CSphIndex_VLND1Ev+0xc5)[0x583cca77d0a5]
/usr/bin/searchd(_ZN13CSphIndex_VLND0Ev+0x9)[0x583cca77d2d9]
/usr/bin/searchd(_ZNK9RtIndex_c17PreallocDiskChunkERK10CSphStringiP17FilenameBuilder_iRN3sph8Vector_TIS0_NS5_13DefaultCopy_TIS0_EENS5_14DefaultRelimitENS5_16DefaultStorage_TIS0_EEEERS0_PKc+0x214)[0x583cca960424]
/usr/bin/searchd(_ZN9RtIndex_c13SaveDiskChunkEbbb+0x67e)[0x583cca95afde]
/usr/bin/searchd(ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB+0x1c)[0x583ccb893c1c]
/usr/bin/searchd(make_fcontext+0x37)[0x583ccb8d5b17]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x000073741DE42520 in /lib/x86_64-linux-gnu/libc.so.6
3# hnswlib::HierarchicalNSW::~HierarchicalNSW() in /usr/share/manticore/modules/lib_manticore_knn.so
4# hnswlib::HierarchicalNSW::~HierarchicalNSW() in /usr/share/manticore/modules/lib_manticore_knn.so
5# knn::HNSWIndex_c::~HNSWIndex_c() in /usr/share/manticore/modules/lib_manticore_knn.so
6# knn::KNN_c::~KNN_c() in /usr/share/manticore/modules/lib_manticore_knn.so
7# CSphIndex_VLN::~CSphIndex_VLN() in /usr/bin/searchd
8# CSphIndex_VLN::~CSphIndex_VLN() in /usr/bin/searchd
9# RtIndex_c::PreallocDiskChunk(CSphString const&, int, FilenameBuilder_i*, sph::Vector_T<CSphString, sph::DefaultCopy_T, sph::DefaultRelimit, sph::DefaultStorage_T >&, CSphString&, char const*) const in /usr/bin/searchd
10# RtIndex_c::SaveDiskChunk(bool, bool, bool) in /usr/bin/searchd
11# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
12# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------

Manticore Search Version:

6.3.0

Operating System Version:

ubuntu 22.04

Have you tried the latest development version?

• No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• closed by author

[chiacy]

I can't reproduce the problem on a brand new instance with exactly the same specs except there's only one index.

Close for now.