Manticore 6.0.0 plain index crashes

[ravelanded]

Describe the bug
Frequent crashes under low to moderate load

To Reproduce
Steps to reproduce the behavior:

1. Have some distributed indexes
2. Load

Expected behavior
It should not crash

Describe the environment:
Manticore 6:
Manticore 6.0.0 8de9df201@230206 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)

Ubuntu 22.04.1 LTS:
Linux mcore2 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Messages from log files:

[Thu Feb  9 16:25:52.327 2023] [37109]  
  
--- crashed SphinxQL request dump ---  
SELECT * FROM cat1095_dist WHERE sphinx_deleted=0 AND deactivated=0 AND blacklisted=0  
AND expires BETWEEN 1675952752 AND 2147483647 AND site_id=1 AND cat_id=1095 AND city_id=1  
AND param174 IN (3444,3446,3448) AND param175 IN (3453,3454) ORDER BY feat DESC, uptime  
DESC, id DESC LIMIT 0,50 OPTION max_matches=5000 FACET cat_id ORDER BY count(*) DESC  
LIMIT 700 FACET adtype ORDER BY count(*) DESC LIMIT 700  
--- request dump end ---  
--- local index:cat1095  
Manticore 6.0.0 8de9df201@230206 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)  
Handling signal 11  
-------------- backtrace begins here ---------------  
Program compiled with Clang 15.0.4  
Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore  
Built on Linux x86_64 (jammy) (cross-compiled)  
Stack bottom = 0x7fece8033a40, thread stack size = 0x20000  
Trying manual backtrace:  
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)  
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7fece8030000, stacksize=0x20000)  
Trying system backtrace:  
begin of system symbols:  
/usr/bin/searchd(_Z12sphBacktraceib 0x22a)[0x559fd7fba35a]  
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi 0x355)[0x559fd7e7cb35]  
/lib/x86_64-linux-gnu/libc.so.6( 0x42520)[0x7fed3824e520]  
/usr/bin/searchd(_ZN21RowidIterator_Union_TIN6common15BlockIterator_iELb0EE9HintRowIDEj 0x2c)[0x559fd82d268c]  
/usr/bin/searchd(_ZN25RowidIterator_Intersect_TI15RowidIterator_iLb0EE17GetNextRowIdBlockER11VecTraits_TIjE 0x23b)[0x559fd82cf2bb]  
/usr/bin/searchd( 0x8c7ef7)[0x559fd7f50ef7]  
/usr/bin/searchd(_ZNK13CSphIndex_VLN21RunFullscanOnIteratorEP15RowidIterator_iRK16CSphQueryContextR19CSphQueryResultMetaRK11VecTraits_TIP15ISphMatchSorterER9CSphMatchibil 0xc2f)[0x559fd7f1a6af]  
/usr/bin/searchd(_ZNK13CSphIndex_VLN9MultiScanER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgsl 0x918)[0x559fd7f1c938]  
/usr/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs 0x2bc)[0x559fd7f26fec]  
/usr/bin/searchd( 0x855816)[0x559fd7ede816]  
/usr/bin/searchd( 0xcfd55d)[0x559fd838655d]  
/usr/bin/searchd(_ZN7Threads4Coro8ExecuteNEiOSt8functionIFvvEE 0x267)[0x559fd8783717]  
/usr/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv 0xb45)[0x559fd7e92655]  
/usr/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii 0x519)[0x559fd7e93df9]  
/usr/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv 0xd4)[0x559fd7e90694]  
/usr/bin/searchd(_Z17HandleMysqlSelectR11RowBuffer_iR15SearchHandler_c 0x1ec)[0x559fd7eb641c]  
/usr/bin/searchd(_Z20HandleMysqlMultiStmtRKN3sph8Vector_TI9SqlStmt_tNS_13DefaultCopy_TIS1_EENS_14DefaultRelimitENS_16DefaultStorage_TIS1_EEEER19CSphQueryResultMetaR11RowBuffer_iRK10CSphString 0x3d1)[0x559fd7eb9801]  
/usr/bin/searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i 0x822)[0x559fd7ec4302]  
/usr/bin/searchd(_Z20ProcessSqlQueryBuddySt4pairIPKciERhR16ISphOutputBuffer 0x47)[0x559fd7e293e7]  
/usr/bin/searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE 0x10b8)[0x559fd7e15658]  
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e 0x43)[0x559fd7e11483]  
/usr/bin/searchd( 0x789081)[0x559fd7e12081]  
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9_ 0x1c)[0x559fd878627c]  
/usr/bin/searchd(make_fcontext 0x37)[0x559fd87a4f37]  
Trying boost backtrace:  
 0# sphBacktrace(int, bool) in /usr/bin/searchd  
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd  
 2# 0x00007FED3824E520 in /lib/x86_64-linux-gnu/libc.so.6  
 3# RowidIterator_Union_T<common::BlockIterator_i, false>::HintRowID(unsigned int) in /usr/bin/searchd  
 4# RowidIterator_Intersect_T<RowidIterator_i, false>::GetNextRowIdBlock(VecTraits_T<unsigned int>&) in /usr/bin/searchd  
 5# 0x0000559FD7F50EF7 in /usr/bin/searchd  
 6# CSphIndex_VLN::RunFullscanOnIterator(RowidIterator_i*, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const in /usr/bin/searchd  
 7# CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const in /usr/bin/searchd  
 8# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /usr/bin/searchd  
 9# 0x0000559FD7EDE816 in /usr/bin/searchd  
10# 0x0000559FD838655D in /usr/bin/searchd  
11# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in /usr/bin/searchd  
12# SearchHandler_c::RunLocalSearches() in /usr/bin/searchd  
13# SearchHandler_c::RunSubset(int, int) in /usr/bin/searchd  
14# SearchHandler_c::RunQueries() in /usr/bin/searchd  
15# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in /usr/bin/searchd  
16# HandleMysqlMultiStmt(sph::Vector_T<SqlStmt_t, sph::DefaultCopy_T<SqlStmt_t>, sph::DefaultRelimit, sph::DefaultStorage_T<SqlStmt_t> > const&, CSphQueryResultMeta&, RowBuffer_i&, CSphString const&) in /usr/bin/searchd  
17# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd  
18# ProcessSqlQueryBuddy(std::pair<char const*, int>, unsigned char&, ISphOutputBuffer&) in /usr/bin/searchd  
19# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd  
20# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /usr/bin/searchd  
21# 0x0000559FD7E12081 in /usr/bin/searchd  
22# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T<unsigned char>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd  
23# make_fcontext in /usr/bin/searchd  

[tomatolog]

could you check your cat1095 index with indextool and upload it into out FTP in case it passed check well?

[ravelanded]

Check passed

checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check passed, 0.0 sec elapsed

Where to upload?

[sanikolaev]

https://manual.manticoresearch.com/Reporting_bugs#Uploading-your-data

[ravelanded]

Done. But the problem is happening with different indexes.

[tomatolog]

you could try to disable secondary indexes and restart daemon to check if crash persists

https://manual.manticoresearch.com/Server_settings/Searchd#secondary_indexes

[sanikolaev]

Thanks. We're reproduced the crash on our side and are working on fixing it.

Disabling secondary indexes by adding:

secondary_indexes = 0

to section "searchd" of the configuration file should help and is recommended as a temporary workaround.

[githubmanticore]

➤ Ilya Kuznetsov commented:

Fixed in d671117