Searchd crash on OS CentOS 7

[masarinetwork]

[Mon Sep 28 10:47:22.018 2020] [25906] WARNING: binlog: update: unexpected tid (index=xxxxxxDelta, indextid=0, logtid=2865, pos=53)
[Mon Sep 28 10:47:22.031 2020] [25906] WARNING: GlobalCrashQueryGetRef: thread-local info is not set! Use ad-hoc

--- crashed invalid query ---

--- request dump end ---
--- local index:
Manticore 3.5.0 1d34c49@200722 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_RE2=1 -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SONAME=libgalera_manticore.so.31 -DSYSCONFDIR=/etc/manticoresearch
Host OS is Linux runner-fa6cab46-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7ffc18072bef, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1195640)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1195640, stack=0xe70000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib 0x90)[0x723f30]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi 0x1b2)[0x592182]
/lib64/libpthread.so.0( 0x109f0)[0x7f4ed754a9f0]
/lib64/libc.so.6( 0x9ec30)[0x7f4ed635ec30]
/usr/bin/searchd(_ZN17AttributePacker_c7SetDataEPKhiR10CSphString 0x2d)[0x8d34bd]
/usr/bin/searchd(_ZN19IndexUpdateHelper_c12Update_BlobsER15UpdateContext_tRbR10CSphString 0x653)[0x6a6c33]
/usr/bin/searchd(ZN13CSphIndex_VLN16UpdateAttributesERK14CSphAttrUpdateiRbR10CSphStringS5 0xbc)[0x6b6aec]
/usr/bin/searchd(_ZNK10RtBinlog_c22ReplayUpdateAttributesEiR14BinlogReader_c 0x33a)[0x89379a]
/usr/bin/searchd(_ZN10RtBinlog_c12ReplayBinlogERK15CSphOrderedHashIP9CSphIndex10CSphString15CSphStrHashFuncLi256EEji 0x4a3)[0x89a853]
/usr/bin/searchd(_ZN10RtBinlog_c6ReplayERK15CSphOrderedHashIP9CSphIndex10CSphString15CSphStrHashFuncLi256EEjPFvvE 0x60)[0x89aba0]
/usr/bin/searchd(_Z15sphReplayBinlogRK15CSphOrderedHashIP9CSphIndex10CSphString15CSphStrHashFuncLi256EEjPFvvE 0x18)[0x89ac98]
/usr/bin/searchd(_Z11ServiceMainiPPc 0x1046)[0x5c1666]
/usr/bin/searchd(main 0x4e)[0x58f4be]
/lib64/libc.so.6(__libc_start_main 0xf0)[0x7f4ed62e0580]
/usr/bin/searchd[0x5917ef]
-------------- backtrace ends here ---------------

Hardware:

HP DL380 Dual Intel(R) Xeon(R) CPU X5690 @ 3.47GHz 12 Cores 24 Threads
RAM 96GB
OS CentOS 7

[tomatolog]

could you provide your data that recreates the crash to our FTP?

I need index files and binlog files.

[masarinetwork]

No problem, we will send it. This crash always on time about 10-11 AM GMT+7 (3 - 4AM UTC)

We use local distributed, index0-4 and delta, delta data using cron every 30 minutes. Main data cron every night at 03 AM GMT+7 (8 PM Prev Day UTC)

When crash, we must recreate all index, and stop start searchd.

[sanikolaev]

@masarinetwork why do you have to recreate your index? Does it get corrupted? Also interesting why you have to stop/start the searchd? Have you disabled the watchdog http://mnt.cr/watchdog ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Feel free to re-open the issue in case it becomes actual.