Crash when performing many INSERT queries mixed with normal traffic

[ericek111]

Describe the bug
When a lot of INSERT requests are being sent, a couple SELECTs is enough to crash the daemon. This is easily and repeatedly reproducible. Possibly related to #1396

Describe the environment:

• Manticore Search from apt: 6.2.12 dc5144d@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
• linux 5.10.0-25-amd64 SMP Debian 5.10.191-1 (2023-08-16)

Messages from log files:

------- FATAL: CRASH DUMP -------
[Sat Sep  2 13:54:59.710 2023] [ 8328]

--- crashed SphinxQL request dump ---
SELECT WEIGHT() AS relevance, * FROM tf_global WHERE 1=1 AND `user` NOT IN(6) AND 
`postStatus` IN('publish') AND MATCH('tapeta') ORDER BY `relevance` DESC LIMIT 0, 
26 
--- request dump end ---
--- local index:tf_global
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=bullseye -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (bullseye) (cross-compiled)
Stack bottom = 0x7f3de8063f60, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f3de8060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55b881f5bcfa]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55b881ddace5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f3e7d6d1140]
/usr/bin/searchd(_ZNK16Expr_GetStored_TILb1EE13GetBlobPackedERK9CSphMatch+0x5c)[0x55b882c4165c]
/usr/bin/searchd(+0xeb914c)[0x55b881e3a14c]
/usr/bin/searchd(+0xeb904c)[0x55b881e3a04c]
/usr/bin/searchd(_Z18MinimizeAggrResultR12AggrResult_tRK9CSphQuerybRKN3sph9StringSetEP14QueryProfile_cPK18CSphFilterSettingsbb+0x6184)[0x55b881dedef4]
/usr/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii+0x1324)[0x55b881df3164]
/usr/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv+0xd4)[0x55b881def064]
/usr/bin/searchd(_Z17HandleMysqlSelectR11RowBuffer_iR15SearchHandler_c+0x14f)[0x55b881e14d8f]
/usr/bin/searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i+0xeb6)[0x55b881e249f6]
/usr/bin/searchd(_Z20ProcessSqlQueryBuddySt4pairIPKciERhR21GenericOutputBuffer_c+0x52)[0x55b881d851a2]
/usr/bin/searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x105d)[0x55b881d6a74d]
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x43)[0x55b881d66503]
/usr/bin/searchd(+0xde60b2)[0x55b881d670b2]
/usr/bin/searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x55b8830a962c]
/usr/bin/searchd(make_fcontext+0x2f)[0x55b8830c99cf]
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /usr/bin/searchd
 1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
 2# 0x00007F3E7D6D1140 in /lib/x86_64-linux-gnu/libpthread.so.0
 3# Expr_GetStored_T<true>::GetBlobPacked(CSphMatch const&) const in /usr/bin/searchd
 4# 0x000055B881E3A14C in /usr/bin/searchd
 5# 0x000055B881E3A04C in /usr/bin/searchd
 6# MinimizeAggrResult(AggrResult_t&, CSphQuery const&, bool, sph::StringSet const&, QueryProfile_c*, CSphFilterSettings const*, bool, bool) in /usr/bin/searchd
 7# SearchHandler_c::RunSubset(int, int) in /usr/bin/searchd
 8# SearchHandler_c::RunQueries() in /usr/bin/searchd
 9# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in /usr/bin/searchd
10# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd
11# ProcessSqlQueryBuddy(std::pair<char const*, int>, unsigned char&, GenericOutputBuffer_c&) in /usr/bin/searchd
12# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd
13# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /usr/bin/searchd
14# 0x000055B881D670B2 in /usr/bin/searchd
15# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
16# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------
--- active threads ---
thd 0 (work_4), proto mysql, state query, command select
--- Totally 2 threads, and 1 client-working threads ---
------- CRASH DUMP END -------

[tomatolog]

could you upload your tf_global index along with insert statements to reproduce this crash locally?

[ericek111]

I have downgraded to 6.0.4 and everything works perfectly fine. This also fixed another issue with increased CPU usage. searchd would suddenly, maybe 30 or so seconds after startup, use more and more CPU cores (while properly serving requests). I thought the index was borked, so I purged it, but it quickly came back after reindexing.

I use a plain real-time index (structure specified in cfg) with some stored fields (for the title and content) and a couple attributes, including a MVA for categories. With all of our content (around 120 000 articles), it's roughly half a gig. Sorry, I don't feel like uploading it publicly. I could send it to your e-mail.

[tomatolog]

you could upload all your data as described at our manual Uploading your data into our write-only S3 storage

It is also unclear you said

I use a plain index with

however initially you mention lot of INSERT statements should be issued parallel to SELECT. I sure there is no way to issue INSERT into plain index.

Coud you provide reproducible example that we could chaeck here localy?

[Eclipsium]

I have a similar situation, but only via json queries
#1458

[tomatolog]

it could be better to post crash log or searchd.log to make sure issues are related

[Eclipsium]

2023-10-04T12:18:03.181633637Z Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2023-10-04T12:18:04.248096473Z Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2023-10-04T12:18:04.565477888Z [Wed Oct  4 12:18:04.565 2023] [1] using config file '/etc/manticoresearch/manticore.conf' (9306 chars)...
2023-10-04T12:18:04.608926835Z [Wed Oct  4 12:18:04.608 2023] [1] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
2023-10-04T12:18:04.608953163Z starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
2023-10-04T12:18:04.608956662Z [Wed Oct  4 12:18:04.608 2023] [1] listening on all interfaces for mysql, port=9306
2023-10-04T12:18:04.608959603Z listening on all interfaces for mysql, port=9306
2023-10-04T12:18:04.609051266Z [Wed Oct  4 12:18:04.608 2023] [1] listening on UNIX socket /var/run/mysqld/mysqld.sock
2023-10-04T12:18:04.609068438Z listening on UNIX socket /var/run/mysqld/mysqld.sock
2023-10-04T12:18:04.609167209Z [Wed Oct  4 12:18:04.609 2023] [1] listening on 10.112.130.5:9312 for sphinx and http(s)
2023-10-04T12:18:04.609183631Z listening on 10.112.130.5:9312 for sphinx and http(s)
2023-10-04T12:18:04.609187132Z [Wed Oct  4 12:18:04.609 2023] [1] listening on all interfaces for sphinx and http(s), port=9308
2023-10-04T12:18:04.609190055Z listening on all interfaces for sphinx and http(s), port=9308
2023-10-04T12:18:04.769504685Z Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2023-10-04T12:18:04.769533511Z Copyright (c) 2001-2016, Andrew Aksyonoff
2023-10-04T12:18:04.769537042Z Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
2023-10-04T12:18:04.769540137Z Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
2023-10-04T12:18:04.769543028Z 
2023-10-04T12:18:04.769545825Z precaching table 'posts_idx'
2023-10-04T12:18:07.191744934Z [Wed Oct  4 12:18:07.191 2023] [74] WARNING: table 'posts_idx': table 'posts_idx': morphology option changed from config has no effect, ignoring
2023-10-04T12:18:07.191773558Z WARNING: table 'posts_idx': table 'posts_idx': morphology option changed from config has no effect, ignoring
2023-10-04T12:18:07.192057114Z [Wed Oct  4 12:18:07.191 2023] [77] binlog: replaying log /var/lib/manticore/binlog/binlog.2085
2023-10-04T12:18:07.192073809Z binlog: replaying log /var/lib/manticore/binlog/binlog.2085
2023-10-04T12:18:08.291960400Z [Wed Oct  4 12:18:08.291 2023] [78] binlog: table posts_idx: recovered from tid 60394328 to tid 60394368
2023-10-04T12:18:08.291990404Z binlog: table posts_idx: recovered from tid 60394328 to tid 60394368
2023-10-04T12:18:08.291995600Z [Wed Oct  4 12:18:08.291 2023] [78] binlog: replay stats: 247 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 tables
2023-10-04T12:18:08.292000370Z binlog: replay stats: 247 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 tables
2023-10-04T12:18:08.292019207Z [Wed Oct  4 12:18:08.291 2023] [78] binlog: finished replaying /var/lib/manticore/binlog/binlog.2085; 257.9 MB in 1.098 sec
2023-10-04T12:18:08.292023675Z binlog: finished replaying /var/lib/manticore/binlog/binlog.2085; 257.9 MB in 1.098 sec
2023-10-04T12:18:08.292028096Z [Wed Oct  4 12:18:08.291 2023] [78] binlog: replaying log /var/lib/manticore/binlog/binlog.2086
2023-10-04T12:18:08.292032287Z binlog: replaying log /var/lib/manticore/binlog/binlog.2086
2023-10-04T12:18:08.591772482Z [Wed Oct  4 12:18:08.591 2023] [79] binlog: table posts_idx: recovered from tid 60394368 to tid 60394430
2023-10-04T12:18:08.591797248Z binlog: table posts_idx: recovered from tid 60394368 to tid 60394430
2023-10-04T12:18:08.591800842Z [Wed Oct  4 12:18:08.591 2023] [79] binlog: replay stats: 62 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 tables
2023-10-04T12:18:08.591803922Z binlog: replay stats: 62 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 1 tables
2023-10-04T12:18:08.591806778Z [Wed Oct  4 12:18:08.591 2023] [79] binlog: finished replaying /var/lib/manticore/binlog/binlog.2086; 71.0 MB in 0.299 sec
2023-10-04T12:18:08.591809944Z binlog: finished replaying /var/lib/manticore/binlog/binlog.2086; 71.0 MB in 0.299 sec
2023-10-04T12:18:08.591812740Z [Wed Oct  4 12:18:08.591 2023] [79] binlog: finished replaying total 2 in 1.399 sec
2023-10-04T12:18:08.591815607Z binlog: finished replaying total 2 in 1.399 sec
2023-10-04T12:18:08.592440029Z [Wed Oct  4 12:18:08.592 2023] [76] prereading 1 tables
2023-10-04T12:18:08.592461385Z prereading 1 tables
2023-10-04T12:18:08.946266429Z [Wed Oct  4 12:18:08.945 2023] [1] accepting connections
2023-10-04T12:18:08.946305889Z accepting connections
2023-10-04T12:18:10.593096864Z [Wed Oct  4 12:18:10.592 2023] [72] [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://10.112.130.5:9312  --threads=16' at http://127.0.0.1:41965
2023-10-04T12:18:10.593136525Z [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://10.112.130.5:9312  --threads=16' at http://127.0.0.1:41965
2023-10-04T12:18:10.749178976Z [Wed Oct  4 12:18:10.749 2023] [79] [BUDDY] Loaded plugins:
2023-10-04T12:18:10.749207589Z [BUDDY] Loaded plugins:
2023-10-04T12:18:10.749210899Z [Wed Oct  4 12:18:10.749 2023] [79] [BUDDY]   core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
2023-10-04T12:18:10.749214408Z [BUDDY]   core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
2023-10-04T12:18:10.749217374Z [Wed Oct  4 12:18:10.749 2023] [79] [BUDDY]   local: 
2023-10-04T12:18:10.749220146Z [BUDDY]   local: 
2023-10-04T12:18:10.749222813Z [Wed Oct  4 12:18:10.749 2023] [79] [BUDDY]   extra: 
2023-10-04T12:18:10.749225646Z [BUDDY]   extra: 
2023-10-04T12:18:24.566188250Z [Wed Oct  4 12:18:24.565 2023] [76] preread 1 tables in 15.974 sec
2023-10-04T12:18:24.566244726Z preread 1 tables in 15.974 sec
2023-10-04T13:24:52.800529970Z [Wed Oct  4 13:24:52.800 2023] [86] rt: table posts_idx: diskchunk 11634(33), segments 10  saved in 2.428252 (3.565889) sec, RAM saved/new 33629190/11186979 ratio 0.750381 (soft limit 100714397, conf limit 134217728)
2023-10-04T13:24:52.800565531Z rt: table posts_idx: diskchunk 11634(33), segments 10  saved in 2.428252 (3.565889) sec, RAM saved/new 33629190/11186979 ratio 0.750381 (soft limit 100714397, conf limit 134217728)
2023-10-04T13:25:10.137270207Z [Wed Oct  4 13:25:10.137 2023] [84] rt: table posts_idx: optimized progressive chunk(s) 1 ( left 32 ) in 17s 336ms
2023-10-04T13:25:10.137315598Z rt: table posts_idx: optimized progressive chunk(s) 1 ( left 32 ) in 17s 336ms
2023-10-04T13:27:35.566982935Z ------- FATAL: CRASH DUMP -------
2023-10-04T13:27:35.567020912Z [Wed Oct  4 13:27:35.396 2023] [    1]
2023-10-04T13:27:35.567026952Z 
2023-10-04T13:27:35.567031877Z --- crashed HTTP request dump ---
2023-10-04T13:27:35.567045026Z {"index": "posts_idx", "query": {"bool": {"must": [{"range": {"uploaded_at": {"gte": 1696424857}}}, {"in": {"any(source_type)": ["ok", "rss", "dzen", "tg", "vk"]}}, {"bool": {"should": [{"query_string": "\"\u043f\u0430\u0442\u0440\u0438\u043e\u0442\""}, {"query_string": "\"\u0441\u043f\u0430\u0441\u043b\u0438\""}, {"query_string": "\"\u043d\u0430\u0433\u0440\u0430\u0434\u0430\""}, {"query_string": "\"\u0434\u0435\u0442\u0438\""}, {"query_string": "\"\u0441\u043f\u0430\u0441\u0435\u043d\u0438\u0435\""}, {"query_string": "\"\u043d\u0430\u0433\u0440\u0430\u0434\u0438\u043b\u0438\""}]}}]}}, "limit": 1000, "offset": 0, "sort": ["posted"], "options": {"max_matches": 50000}, "highlight": {"limit": 50000}}
2023-10-04T13:27:35.567058074Z --- request dump end ---
2023-10-04T13:27:35.567062635Z --- local index:posts_idx
2023-10-04T13:27:35.567066966Z Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2023-10-04T13:27:35.567071924Z Handling signal 11
2023-10-04T13:27:35.567076996Z Crash!!! Handling signal 11
2023-10-04T13:27:35.567081873Z -------------- backtrace begins here ---------------
2023-10-04T13:27:35.567086640Z Program compiled with Clang 15.0.7
2023-10-04T13:27:35.567091588Z Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
2023-10-04T13:27:35.567098737Z Built on Linux x86_64 (jammy) (cross-compiled)
2023-10-04T13:27:35.567103326Z Stack bottom = 0x7fb6cc078070, thread stack size = 0x20000
2023-10-04T13:27:35.567126134Z Trying manual backtrace:
2023-10-04T13:27:35.608826525Z Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
2023-10-04T13:27:35.608863059Z Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7fb6cc080000, stacksize=0x20000)
2023-10-04T13:27:35.608868162Z Trying system backtrace:
2023-10-04T13:27:36.330752819Z begin of system symbols:
2023-10-04T13:27:36.378583510Z searchd(_Z12sphBacktraceib+0x22a)[0x5599d39839fa]
2023-10-04T13:27:36.378898502Z searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x5599d38022b5]
2023-10-04T13:27:36.378920664Z /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fb75fbe7520]
2023-10-04T13:27:36.379413883Z searchd(_ZNK16Expr_Highlight_c22RearrangeFetchedFieldsERK13DocstoreDoc_t+0x1bc)[0x5599d468fdcc]
2023-10-04T13:27:36.379770761Z searchd(_ZNK16Expr_Highlight_c10StringEvalERK9CSphMatchPPKh+0x1b1)[0x5599d468f8e1]
2023-10-04T13:27:36.423435691Z searchd(_ZNK8ISphExpr16StringEvalPackedERK9CSphMatch+0x20)[0x5599d4520ab0]
2023-10-04T13:27:36.423881544Z searchd(+0xeb994c)[0x5599d386194c]
2023-10-04T13:27:36.424351950Z searchd(+0xeb984c)[0x5599d386184c]
2023-10-04T13:27:36.424717145Z searchd(_Z18MinimizeAggrResultR12AggrResult_tRK9CSphQuerybRKN3sph9StringSetEP14QueryProfile_cPK18CSphFilterSettingsbb+0x61f4)[0x5599d3815564]
2023-10-04T13:27:36.425110414Z searchd(_ZN15SearchHandler_c9RunSubsetEii+0x1324)[0x5599d381a7f4]
2023-10-04T13:27:36.425447344Z searchd(_ZN15SearchHandler_c10RunQueriesEv+0xd4)[0x5599d38166e4]
2023-10-04T13:27:36.425817006Z searchd(_ZN19HttpSearchHandler_c7ProcessEv+0xaa)[0x5599d3744eda]
2023-10-04T13:27:36.426163214Z searchd(_Z16ProcessHttpQueryR12CharStream_cRSt4pairIPKciER15CSphOrderedHashI10CSphStringS7_15CSphStrHashFuncLi256EERN3sph8Vector_TIhNSB_13DefaultCopy_TIhEENSB_14DefaultRelimitENSB_16DefaultStorage_TIhEEEEb11http_method+0x5fa)[0x5599d373cfda]
2023-10-04T13:27:36.426544878Z searchd(_Z21ProcessHttpQueryBuddyR12CharStream_cR15CSphOrderedHashI10CSphStringS2_15CSphStrHashFuncLi256EERN3sph8Vector_TIhNS6_13DefaultCopy_TIhEENS6_14DefaultRelimitENS6_16DefaultStorage_TIhEEEEb11http_method+0x45)[0x5599d37abc45]
2023-10-04T13:27:36.426847609Z searchd(_ZN19HttpRequestParser_c17ProcessClientHttpER21AsyncNetInputBuffer_cRN3sph8Vector_TIhNS2_13DefaultCopy_TIhEENS2_14DefaultRelimitENS2_16DefaultStorage_TIhEEEE+0x158)[0x5599d373d938]
2023-10-04T13:27:36.427346696Z searchd(_Z9HttpServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x3c0)[0x5599d3790ab0]
2023-10-04T13:27:36.427582740Z searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0xc7)[0x5599d378df97]
2023-10-04T13:27:36.428456604Z searchd(+0xde6af2)[0x5599d378eaf2]
2023-10-04T13:27:36.429152496Z searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_+0x1c)[0x5599d4ad2dac]
2023-10-04T13:27:36.429597111Z searchd(make_fcontext+0x37)[0x5599d4af3167]
2023-10-04T13:27:36.429641336Z Trying boost backtrace:
2023-10-04T13:27:36.464686556Z  0# sphBacktrace(int, bool) in searchd
2023-10-04T13:27:36.464726893Z  1# CrashLogger::HandleCrash(int) in searchd
2023-10-04T13:27:36.464738397Z  2# 0x00007FB75FBE7520 in /lib/x86_64-linux-gnu/libc.so.6
2023-10-04T13:27:36.464743549Z  3# Expr_Highlight_c::RearrangeFetchedFields(DocstoreDoc_t const&) const in searchd
2023-10-04T13:27:36.464749720Z  4# Expr_Highlight_c::StringEval(CSphMatch const&, unsigned char const**) const in searchd
2023-10-04T13:27:36.464754789Z  5# ISphExpr::StringEvalPacked(CSphMatch const&) const in searchd
2023-10-04T13:27:36.464759513Z  6# 0x00005599D386194C in searchd
2023-10-04T13:27:36.464764083Z  7# 0x00005599D386184C in searchd
2023-10-04T13:27:36.464768714Z  8# MinimizeAggrResult(AggrResult_t&, CSphQuery const&, bool, sph::StringSet const&, QueryProfile_c*, CSphFilterSettings const*, bool, bool) in searchd
2023-10-04T13:27:36.464774039Z  9# SearchHandler_c::RunSubset(int, int) in searchd
2023-10-04T13:27:36.464778822Z 10# SearchHandler_c::RunQueries() in searchd
2023-10-04T13:27:36.464783508Z 11# HttpSearchHandler_c::Process() in searchd
2023-10-04T13:27:36.464788431Z 12# ProcessHttpQuery(CharStream_c&, std::pair<char const*, int>&, CSphOrderedHash<CSphString, CSphString, CSphStrHashFunc, 256>&, sph::Vector_T<unsigned char, sph::DefaultCopy_T<unsigned char>, sph::DefaultRelimit, sph::DefaultStorage_T<unsigned char> >&, bool, http_method) in searchd
2023-10-04T13:27:36.464795642Z 13# ProcessHttpQueryBuddy(CharStream_c&, CSphOrderedHash<CSphString, CSphString, CSphStrHashFunc, 256>&, sph::Vector_T<unsigned char, sph::DefaultCopy_T<unsigned char>, sph::DefaultRelimit, sph::DefaultStorage_T<unsigned char> >&, bool, http_method) in searchd
2023-10-04T13:27:36.464800653Z 14# HttpRequestParser_c::ProcessClientHttp(AsyncNetInputBuffer_c&, sph::Vector_T<unsigned char, sph::DefaultCopy_T<unsigned char>, sph::DefaultRelimit, sph::DefaultStorage_T<unsigned char> >&) in searchd
2023-10-04T13:27:36.464804282Z 15# HttpServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in searchd
2023-10-04T13:27:36.464807546Z 16# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in searchd
2023-10-04T13:27:36.464811028Z 17# 0x00005599D378EAF2 in searchd
2023-10-04T13:27:36.464814132Z 18# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd
2023-10-04T13:27:36.464817767Z 19# make_fcontext in searchd
2023-10-04T13:27:36.464826211Z 
2023-10-04T13:27:36.464830171Z -------------- backtrace ends here ---------------
2023-10-04T13:27:36.464836242Z Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
2023-10-04T13:27:36.464856386Z and attach there:
2023-10-04T13:27:36.464859504Z a) searchd log, b) searchd binary, c) searchd symbols.
2023-10-04T13:27:36.464862477Z Look into the chapter 'Reporting bugs' in the manual
2023-10-04T13:27:36.464865531Z (https://manual.manticoresearch.com/Reporting_bugs)
2023-10-04T13:27:36.464868560Z Dump with GDB via watchdog
2023-10-04T13:27:39.467094313Z --- active threads ---
2023-10-04T13:27:39.467150197Z thd 0 (work_12), proto http, state net_read, command -
2023-10-04T13:27:39.467155116Z --- Totally 2 threads, and 1 client-working threads ---
2023-10-04T13:27:39.467159452Z ------- CRASH DUMP END -------

[tomatolog]

it is not the same crash as I see the highlighter related code but original crash was related to stored fields.
However we did not the data that reproduces the crash from the topic starter.

Could you upload your index mentioned at the crash log to reproduce the crash locally?

I see this index at the crash log posts_idx

[Eclipsium]

I posted a script that reproduces this crash on version 6.2.12.
The strangest thing is that the crash can happen in completely random situations. If you repeat the request where manticore crashes, it works correctly after a restart

https://github.com/Eclipsium/manticore_crash

docker compose up

index config:

 "CREATE TABLE posts_idx ("
    "id BIGINT, "
    "posted TIMESTAMP, "
    "uploaded_at TIMESTAMP, "
    "title TEXT, "
    "content TEXT, "
    "source_type STRING,  "
    "source_id INTEGER, "
    "is_blogger BOOL) "
    "index_exact_words = '1' "
    "morphology = 'lemmatize_ru_all, lemmatize_en_all' "
    "charset_table = '0..9, english, russian, _')"

[sanikolaev]

I posted a script that reproduces this crash on version 6.2.12.

The crash was confirmed in 6.2.12 with help of the script, but we couldn't reproduce it in the dev version.

@ericek111 can you check if the dev version helps in your case too?

[sanikolaev]

we couldn't reproduce it in the dev version

Since the dev version is always changing, it's worth to add that the version I tested on was manticoresearch/manticore:dev-6.2.13-f94555a