Searchd crushes on CentOS 7

[kolu4iy]

Indextool says no problem, but searchd sometimes crushes with this request:

------- FATAL: CRASH DUMP -------
[Mon Dec 23 16:40:42.495 2019] [16774]

--- crashed SphinxAPI request dump ---
AAABHgAACSwAAAAAAAAABAAAABAAAAAAAAAD6AAAAAYAAAAIAAAAUXN1bSgoc3VtX2lkZiszKihtaW5faGl0X3Bv
cz09MSkrMioobWluX2hpdF9wb3M9PTIpKzQqZXhhY3RfaGl0KSp1c2VyX3dlaWdodCkqMTAwMAAAAAAA
AAAAAAAAjShAKG5hbWUsd29yZGZvcm1zX25hbWUsY29kZSxhcnRpY2xlczEsYXJ0aWNsZXMsYWNhdF9u
YW1lLHRlY2RvY19uYW1lKSAo0JrQntCS0KDQmNCaINCR0JDQk9CQ0JbQndCY0JrQkCkgfCAo0JrQntCS
0KDQmNCaINCR0JDQk9CQ0JbQndCY0JrQkCkgKQAAAAAAAAAhY2F0YWxvZyxjYXRhbG9nX21vcnBo
LG1hcmtldHBsYWNlAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAACAAAAFHByb3BlcnR5X21zZWN0aW9uX2lk
AAAAAAAAAAQAAAAAAAAGMwAAAAAAABlDAAAAAAAAGUIAAAAAAAAZRAAAAAAAAAAScHJvcGVydHlfaWJs
b2NrX2lkAAAAAAAAAAEAAAAAAAAAMQAAAAAAAAAEAAAAEVBST1BFUlRZX0FQXzExMTY2AAAD6AAAAAtA
Z3JvdXAgZGVzYwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAABG5hbWUAAAABAAAA
AAAAAAAAAAAkaWQsQGdyb3VwYnksQGNvdW50LFBST1BFUlRZX0FQXzExMTY2AAAAAAAAAAAAAAAAAAAA
AAAAACAAAAAAAAAD6AAAAAYAAAAIAAAAUXN1bSgoc3VtX2lkZiszKihtaW5faGl0X3Bvcz09MSkrMioo
bWluX2hpdF9wb3M9PTIpKzQqZXhhY3RfaGl0KSp1c2VyX3dlaWdodCkqMTAwMAAAAAAAAAAAAAAAjShA
KG5hbWUsd29yZGZvcm1zX25hbWUsY29kZSxhcnRpY2xlczEsYXJ0aWNsZXMsYWNhdF9uYW1lLHRl
Y2RvY19uYW1lKSAo0JrQntCS0KDQmNCaINCR0JDQk9CQ0JbQndCY0JrQkCkgfCAo0JrQntCS0KDQmNCa
INCR0JDQk9CQ0JbQndCY0JrQkCkgKQAAAAAAAAAhY2F0YWxvZyxjYXRhbG9nX21vcnBoLG1hcmtldHBs
YWNlAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAACAAAAFHByb3BlcnR5X21zZWN0aW9uX2lkAAAAAAAAAAQA
AAAAAAAGMwAAAAAAABlDAAAAAAAAGUIAAAAAAAAZRAAAAAAAAAAScHJvcGVydHlfaWJsb2NrX2lk
AAAAAAAAAAEAAAAAAAAAMQAAAAAAAAAEAAAAEVBST1BFUlRZX0FQXzEwMzY2AAAD6AAAAAtAZ3JvdXAg
ZGVzYwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAABG5hbWUAAAABAAAAAAAAAAAAAAAk
aWQsQGdyb3VwYnksQGNvdW50LFBST1BFUlRZX0FQXzEwMzY2AAAAAAAAAAAAAAAAAAAAAAAAADAAAAAA
AAAD6AAAAAYAAAAIAAAAUXN1bSgoc3VtX2lkZiszKihtaW5faGl0X3Bvcz09MSkrMioobWluX2hp
dF9wb3M9PTIpKzQqZXhhY3RfaGl0KSp1c2VyX3dlaWdodCkqMTAwMAAAAAAAAAAAAAAAjShAKG5hbWUs
d29yZGZvcm1zX25hbWUsY29kZSxhcnRpY2xlczEsYXJ0aWNsZXMsYWNhdF9uYW1lLHRlY2RvY19uYW1l
KSAo0JrQntCS0KDQmNCaINCR0JDQk9CQ0JbQndCY0JrQkCkgfCAo0JrQntCS0KDQmNCaINCR0JDQk9CQ
0JbQndCY0JrQkCkgKQAAAAAAAAAhY2F0YWxvZyxjYXRhbG9nX21vcnBoLG1hcmtldHBsYWNlAAAA
AQAAAAAAAAAAAAAAAAAAAAAAAAACAAAAFHByb3BlcnR5X21zZWN0aW9uX2lkAAAAAAAAAAQAAAAAAAAG
MwAAAAAAABlDAAAAAAAAGUIAAAAAAAAZRAAAAAAAAAAScHJvcGVydHlfaWJsb2NrX2lkAAAAAAAAAAEA
AAAAAAAAMQAAAAAAAAAEAAAAEVBST1BFUlRZX0FQXzAwMEQ4AAAD6AAAAAtAZ3JvdXAgZGVzYwAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAABG5hbWUAAAABAAAAAAAAAAAAAAAkaWQsQGdy
b3VwYnksQGNvdW50LFBST1BFUlRZX0FQXzAwMEQ4AAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAD6AAA
AAYAAAAIAAAAUXN1bSgoc3VtX2lkZiszKihtaW5faGl0X3Bvcz09MSkrMioobWluX2hpdF9wb3M9PTIp
KzQqZXhhY3RfaGl0KSp1c2VyX3dlaWdodCkqMTAwMAAAAAAAAAAAAAAAjShAKG5hbWUsd29yZGZvcm1z
X25hbWUsY29kZSxhcnRpY2xlczEsYXJ0aWNsZXMsYWNhdF9uYW1lLHRlY2RvY19uYW1lKSAo0JrQ
ntCS0KDQmNCaINCR0JDQk9CQ0JbQndCY0JrQkCkgfCAo0JrQntCS0KDQmNCaINCR0JDQk9CQ0JbQndCY
0JrQkCkgKQAAAAAAAAAhY2F0YWxvZyxjYXRhbG9nX21vcnBoLG1hcmtldHBsYWNlAAAAAQAAAAAAAAAA
AAAAAAAAAAAAAAACAAAAFHByb3BlcnR5X21zZWN0aW9uX2lkAAAAAAAAAAQAAAAAAAAGMwAAAAAAABlD
AAAAAAAAGUIAAAAAAAAZRAAAAAAAAAAScHJvcGVydHlfaWJsb2NrX2lkAAAAAAAAAAEAAAAAAAAA
MQAAAAAAAAAEAAAAEVBST1BFUlRZX0FQXzExMTY1AAAD6AAAAAtAZ3JvdXAgZGVzYwAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAABAAAABG5hbWUAAAABAAAAAAAAAAAAAAAkaWQsQGdyb3VwYnksQGNv
dW50LFBST1BFUlRZX0FQXzExMTY1AAAAAAAAAAAAAAAAAAAAAA==
--- request dump end ---
--- local index:
Manticore 3.2.2 62ea5ff@191220 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=etc/manticoresearch
Host OS is Linux runner-72989761-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f22fe5b2dff, thread stack size = 0x200000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7f22ec05b7f0)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x7f22ec05b7f0, stack=0x7f22fe5c0000, stacksize=0x200000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x90)[0x6f20f0]
/usr/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi+0x1ce)[0x56d3be]
/lib64/libpthread.so.0(+0xf5e0)[0x7f23c60965e0]
/lib64/libc.so.6(cfree+0x1c)[0x7f23c48334dc]
/usr/bin/searchd(_ZN16CSphSchemaHelper15FreeDataSpecialEP9CSphMatchRKN3sph8Vector_TIiNS2_13DefaultCopy_TIiEENS2_14DefaultRelimitENS2_16DefaultStorage_TIiEEEE+0x4a)[0x644a0a]
/usr/bin/searchd(_ZN15CSphQueryResultD1Ev+0x3e)[0x667a8e]
/usr/bin/searchd(_ZN15SearchHandler_cD1Ev+0x32f)[0x58d2ff]
/usr/bin/searchd(_Z19HandleCommandSearchR20CachedOutputBuffer_ctR13InputBuffer_cRN7Threads9ThdDesc_tE+0x311)[0x5c6471]
/usr/bin/searchd[0x5c9aa4]
/usr/bin/searchd(_ZN11ThdJobAPI_t4CallEv+0x24e)[0x5ca4ce]
/usr/bin/searchd(_ZN11CSphThdPool4TickEPv+0xa0)[0x6fc8a0]
/usr/bin/searchd(_Z20sphThreadProcWrapperPv+0x25)[0x6f8235]
/lib64/libpthread.so.0(+0x7e25)[0x7f23c608ee25]
/lib64/libc.so.6(clone+0x6d)[0x7f23c48ab34d]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)
[Mon Dec 23 16:40:42.704 2019] [16044] watchdog: got USR1, performing dump of child's stack
Will run gdb on /usr/bin/searchd, pid 16774
--- 1 active threads ---
thd 0, proto sphinxapi, state query, command search
------- CRASH DUMP END -------

Indexes are about 5 Gb, config also not so small.

[tomatolog]

you could upload your indexes and config at write-only FTP for customer data:

ftp: dev.manticoresearch.com
user: manticorebugs
pass: shithappens

For issues please use for the folder name github-issue-XXX, where XXX is the issue number.

As BT from crash log seems ordinary for me and without data there crash could be reproduced it is hard to continue investigation.

[kolu4iy]

I'm upload config, logs and indexes. Config file is dynamically generated and this file is currently static representation of it. Mysql data for indexes have ~150 Gb and i'm not upload it. Crushes occur only in case of high load on the sphinx, but only on this type of queries. Thank you.

[tomatolog]

at your marketplace index there is two attributes with same name and different types

FAILED, duplicate attributes name property_ap_length for columns: float at 32@608, mva64 at blob@7      

that is wrong and cause the crash. I just pushed code into master for indextool to check and report such case as index error. You need to change your index declaration and reindex data into that index.

[tomatolog]

I also fixed attributes with duplicated names at index definition at e0baf73

Now indexer should also reports error for such configs