Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CRASH on specific select #364

Open
hgfdd opened this issue Jul 1, 2020 · 7 comments
Open

CRASH on specific select #364

hgfdd opened this issue Jul 1, 2020 · 7 comments
Labels
bug

Comments

@hgfdd
Copy link

hgfdd commented Jul 1, 2020 •
edited by githubmanticore
Loading

Describe the environment

Manticore Search version:

Manticore 3.2.2 afd6046@191226 release

OS version:

Linux aspat-cloud 4.4.5-1.el6.elrepo.x86_64 #1 SMP Thu Mar 10 13:51:50 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Describe the problem:

In specific select queries it crashes. Indextool say the index is fine. Also i can't share the index data.

Steps to reproduce:

Execute the following query several times:

SELECT id, weight() AS weight FROM mdb_morf_archive_idx WHERE MATCH('@*  (((Marsh NEAR/5 McLennan) | (страховой NEAR/5 брокер NEAR/5 Marsh))) @alwaysmatch everyfragment')  AND is_cut = 1 and source_tag_id <> 4010 and reprocess_status <> 2 and start_date  >= 20200522000000 and start_date <= 20200622142137 GROUP BY id WITHIN    
GROUP ORDER  BY weight DESC  ORDER BY start_date desc LIMIT 0, 1000000 OPTION ranker=expr('top(lccs*user_weight)'), max_matches=1000000

Messsages from log files:

------- FATAL: CRASH DUMP -------   
[Wed Jul  1 10:01:54.867 2020] [13152]   
   
--- crashed SphinxQL request dump ---   
SELECT id, weight() AS weight FROM mdb_morf_archive_idx WHERE MATCH('@*  (((Marsh    
NEAR/5   
   
McLennan) | (страховой NEAR/5 брокер NEAR/5 Marsh))) @alwaysmatch    
everyfragment')  AND is_cut = 1 and source_tag_id <> 4010 and reprocess_status <>    
2 and start_date  >= 20200522000000 and start_date <= 20200622142137 GROUP BY id WITHIN    
GROUP ORDER  BY weight DESC  ORDER BY start_date desc LIMIT 0, 1000000 OPTION ranker=expr('top(lccs*user_weight)'),   
   max_matches=1000000   
--- request dump end ---   
--- local index:   
Manticore 3.2.2 afd6046@191226 release   
Handling signal 11   
-------------- backtrace begins here ---------------   
Program compiled with 4.8.2   
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=1 -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_ODBC=1 -DWITH_PGSQL=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON   
Host OS is Linux centos6-dev 2.6.32-431.5.1.el6.x86_64 <a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a> SMP Wed Feb 12 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux   
Stack bottom = 0x7f553f50ce7f, thread stack size = 0x100000   
Trying manual backtrace:   
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7f536c0bedf0)   
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x7f536c0bedf0, stack=0x7f553f510000, stacksize=0x100000)   
Trying system backtrace:   
begin of system symbols:   
/usr/local/bin/searchd(_Z12sphBacktraceib 0x90)[0x65da00]   
/usr/local/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi 0x1ce)[0x51fe5e]   
/lib64/libpthread.so.0[0x3a3800f7e0]   
/lib64/libc.so.6(cfree 0x1c)[0x3a37c7ba1c]   
/usr/local/bin/searchd(_ZN10CSphString8SafeFreeEv 0x1a)[0x57dd6a]   
/usr/local/bin/searchd(_ZN9CSphQueryD1Ev 0xbe)[0x58bf4e]   
/usr/local/bin/searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR16ISphOutputBufferRhRN7Threads9ThdDesc_tE 0x29f)[0x59fb0f]   
/usr/local/bin/searchd[0x57b4e5]   
/usr/local/bin/searchd(_ZN10ThdJobQL_t4CallEv 0x31e)[0x57be8e]   
/usr/local/bin/searchd(_ZN11CSphThdPool4TickEPv 0xa0)[0x6688a0]   
/usr/local/bin/searchd(_Z20sphThreadProcWrapperPv 0x25)[0x664235]   
/lib64/libpthread.so.0[0x3a38007aa1]   
/lib64/libc.so.6(clone 0x6d)[0x3a37ce8aad]   
-------------- backtrace ends here ---------------   
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)   
and attach there:   
a) searchd log, b) searchd binary, c) searchd symbols.   
Look into the chapter 'Reporting bugs' in the documentation   
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)   
[Wed Jul  1 10:01:55.097 2020] [26158] watchdog: got USR1, performing dump of child's stack   
Will run gdb on /usr/local/bin/searchd, pid 13152   
[New LWP 19393]   
[New LWP 13420]   
[New LWP 13419]   
[New LWP 13417]   
[New LWP 13416]   
[New LWP 13415]   
[New LWP 13414]   
[New LWP 13413]   
[New LWP 13412]   
[New LWP 13411]   
[New LWP 13410]   
[New LWP 13409]   
[New LWP 13408]   
[New LWP 13407]   
[New LWP 13406]   
[New LWP 13405]   
[New LWP 13404]   
[New LWP 13403]   
[New LWP 13402]   
[New LWP 13401]   
[New LWP 13400]   
[New LWP 13399]   
[New LWP 13398]   
[New LWP 13397]   
[New LWP 13396]   
[New LWP 13395]   
[New LWP 13394]   
[New LWP 13393]   
[New LWP 13392]   
[New LWP 13391]   
[New LWP 13390]   
[New LWP 13389]   
[New LWP 13388]   
[New LWP 13387]   
[New LWP 13386]   
[New LWP 13385]   
[New LWP 13384]   
[New LWP 13383]   
[New LWP 13382]   
[New LWP 13381]   
[New LWP 13380]   
[New LWP 13379]   
[New LWP 13378]   
[New LWP 13377]   
[New LWP 13376]   
[New LWP 13375]   
[New LWP 13374]   
[New LWP 13373]   
[New LWP 13372]   
[New LWP 13371]   
[New LWP 13370]   
[New LWP 13369]   
[New LWP 13368]   
[New LWP 13367]   
[New LWP 13366]   
[New LWP 13365]   
[New LWP 13364]   
[New LWP 13363]   
[New LWP 13362]   
[New LWP 13361]   
[New LWP 13360]   
[New LWP 13359]   
[New LWP 13358]   
[New LWP 13357]   
[New LWP 13356]   
[New LWP 13355]   
[New LWP 13354]   
[New LWP 13353]   
[New LWP 13352]   
[New LWP 13351]   
[New LWP 13348]   
[New LWP 13347]   
[New LWP 13346]   
[New LWP 13345]   
[New LWP 13344]   
[New LWP 13343]   
[New LWP 13342]   
[New LWP 13341]   
[New LWP 13340]   
[New LWP 13339]   
[New LWP 13338]   
[New LWP 13337]   
[New LWP 13336]   
[New LWP 13335]   
[New LWP 13334]   
[New LWP 13333]   
[New LWP 13332]   
[New LWP 13331]   
[New LWP 13330]   
[New LWP 13329]   
[New LWP 13328]   
[New LWP 13327]   
[New LWP 13326]   
[New LWP 13325]   
[New LWP 13324]   
[New LWP 13323]   
[New LWP 13322]   
[New LWP 13321]   
[New LWP 13320]   
[New LWP 13319]   
[New LWP 13318]   
[New LWP 13317]   
[New LWP 13316]   
[New LWP 13250]   
[New LWP 13157]   
[Thread debugging using libthread_db enabled]   
0x0000003a37ce1523 in select () from /lib64/libc.so.6   
  106 Thread 0x7f6351659700 (LWP 13157)  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  105 Thread 0x7f6350269700 (LWP 13250)  0x000000000060bd7e in unsigned char PrereadMapping<unsigned char>(char const*, char const*, bool, bool, CSphBufferTrait<unsigned char>&) ()   
  104 Thread 0x7f55428d9700 (LWP 13316)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  103 Thread 0x7f55427d5700 (LWP 13317)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  102 Thread 0x7f55426d1700 (LWP 13318)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  101 Thread 0x7f55425cd700 (LWP 13319)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  100 Thread 0x7f55424c9700 (LWP 13320)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  99 Thread 0x7f55423c5700 (LWP 13321)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  98 Thread 0x7f55422c1700 (LWP 13322)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  97 Thread 0x7f55421bd700 (LWP 13323)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  96 Thread 0x7f55420b9700 (LWP 13324)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  95 Thread 0x7f5541fb5700 (LWP 13325)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  94 Thread 0x7f5541eb1700 (LWP 13326)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  93 Thread 0x7f5541dad700 (LWP 13327)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  92 Thread 0x7f5541ca9700 (LWP 13328)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  91 Thread 0x7f5541ba5700 (LWP 13329)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  90 Thread 0x7f5541aa1700 (LWP 13330)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  89 Thread 0x7f554199d700 (LWP 13331)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  88 Thread 0x7f5541899700 (LWP 13332)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  87 Thread 0x7f5541795700 (LWP 13333)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  86 Thread 0x7f5541691700 (LWP 13334)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  85 Thread 0x7f554158d700 (LWP 13335)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  84 Thread 0x7f5541489700 (LWP 13336)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  83 Thread 0x7f5541385700 (LWP 13337)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  82 Thread 0x7f5541281700 (LWP 13338)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  81 Thread 0x7f554117d700 (LWP 13339)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  80 Thread 0x7f5541079700 (LWP 13340)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  79 Thread 0x7f5540f75700 (LWP 13341)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  78 Thread 0x7f5540e71700 (LWP 13342)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  77 Thread 0x7f5540d6d700 (LWP 13343)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  76 Thread 0x7f5540c69700 (LWP 13344)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  75 Thread 0x7f5540b65700 (LWP 13345)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  74 Thread 0x7f5540a61700 (LWP 13346)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  73 Thread 0x7f554095d700 (LWP 13347)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  72 Thread 0x7f5540859700 (LWP 13348)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  71 Thread 0x7f5540755700 (LWP 13351)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  70 Thread 0x7f5540651700 (LWP 13352)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  69 Thread 0x7f554054d700 (LWP 13353)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  68 Thread 0x7f5540449700 (LWP 13354)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  67 Thread 0x7f5540345700 (LWP 13355)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  66 Thread 0x7f5540241700 (LWP 13356)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  65 Thread 0x7f554013d700 (LWP 13357)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  64 Thread 0x7f5540039700 (LWP 13358)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  63 Thread 0x7f553ff35700 (LWP 13359)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  62 Thread 0x7f553fe31700 (LWP 13360)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  61 Thread 0x7f553fd2d700 (LWP 13361)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  60 Thread 0x7f553fc29700 (LWP 13362)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  59 Thread 0x7f553fb25700 (LWP 13363)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  58 Thread 0x7f553fa21700 (LWP 13364)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  57 Thread 0x7f553f91d700 (LWP 13365)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  56 Thread 0x7f553f819700 (LWP 13366)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  55 Thread 0x7f553f715700 (LWP 13367)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  54 Thread 0x7f553f611700 (LWP 13368)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  53 Thread 0x7f553f50d700 (LWP 13369)  0x0000003a37ce1523 in select () from /lib64/libc.so.6   
  52 Thread 0x7f553f409700 (LWP 13370)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  51 Thread 0x7f553f305700 (LWP 13371)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  50 Thread 0x7f553f201700 (LWP 13372)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  49 Thread 0x7f553f0fd700 (LWP 13373)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  48 Thread 0x7f553eff9700 (LWP 13374)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  47 Thread 0x7f553eef5700 (LWP 13375)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  46 Thread 0x7f553edf1700 (LWP 13376)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  45 Thread 0x7f553eced700 (LWP 13377)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  44 Thread 0x7f553ebe9700 (LWP 13378)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  43 Thread 0x7f553eae5700 (LWP 13379)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  42 Thread 0x7f553e9e1700 (LWP 13380)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  41 Thread 0x7f553e8dd700 (LWP 13381)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  40 Thread 0x7f553e7d9700 (LWP 13382)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  39 Thread 0x7f553e6d5700 (LWP 13383)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  38 Thread 0x7f553e5d1700 (LWP 13384)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  37 Thread 0x7f553e4cd700 (LWP 13385)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  36 Thread 0x7f553e3c9700 (LWP 13386)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  35 Thread 0x7f553e2c5700 (LWP 13387)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  34 Thread 0x7f553e1c1700 (LWP 13388)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  33 Thread 0x7f553e0bd700 (LWP 13389)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  32 Thread 0x7f553dfb9700 (LWP 13390)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  31 Thread 0x7f553deb5700 (LWP 13391)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  30 Thread 0x7f553ddb1700 (LWP 13392)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  29 Thread 0x7f553dcad700 (LWP 13393)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  28 Thread 0x7f553dba9700 (LWP 13394)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  27 Thread 0x7f553daa5700 (LWP 13395)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  26 Thread 0x7f553d9a1700 (LWP 13396)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  25 Thread 0x7f553d89d700 (LWP 13397)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  24 Thread 0x7f553d799700 (LWP 13398)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  23 Thread 0x7f553d695700 (LWP 13399)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  22 Thread 0x7f553d591700 (LWP 13400)  0x0000003a37c89cec in _wordcopy_fwd_aligned () from /lib64/libc.so.6   
  21 Thread 0x7f553d48d700 (LWP 13401)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  20 Thread 0x7f553d389700 (LWP 13402)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  19 Thread 0x7f553d285700 (LWP 13403)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  18 Thread 0x7f553d181700 (LWP 13404)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  17 Thread 0x7f553d07d700 (LWP 13405)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  16 Thread 0x7f553cf79700 (LWP 13406)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  15 Thread 0x7f553ce75700 (LWP 13407)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  14 Thread 0x7f553cd71700 (LWP 13408)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  13 Thread 0x7f553cc6d700 (LWP 13409)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  12 Thread 0x7f553cb69700 (LWP 13410)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  11 Thread 0x7f553ca65700 (LWP 13411)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  10 Thread 0x7f553c961700 (LWP 13412)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  9 Thread 0x7f553c85d700 (LWP 13413)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  8 Thread 0x7f553c759700 (LWP 13414)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  7 Thread 0x7f553c655700 (LWP 13415)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  6 Thread 0x7f553c551700 (LWP 13416)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  5 Thread 0x7f553c44d700 (LWP 13417)  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  4 Thread 0x7f553c147700 (LWP 13419)  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
  3 Thread 0x7f5537fff700 (LWP 13420)  0x0000003a37ce90a3 in epoll_wait () from /lib64/libc.so.6   
  2 Thread 0x7f5537e07700 (LWP 19393)  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
* 1 Thread 0x7f635165b720 (LWP 13152)  0x0000003a37ce1523 in select () from /lib64/libc.so.6   
   
Thread 106 (Thread 0x7f6351659700 (LWP 13157)):   
#0  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664d86 in AutoEvent_T<true>::WaitEvent(int) ()   
#2  0x00007f63518d27ea in EventTick (pArg=0x7f6351afdbc0) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:512   
#3  EventLoop (pArg=0x7f6351afdbc0) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:452   
#4  LazyJobs_c::WorkerFunc (pArg=0x7f6351afdbc0) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:581   
#5  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#6  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#7  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 105 (Thread 0x7f6350269700 (LWP 13250)):   
#0  0x000000000060bd7e in unsigned char PrereadMapping<unsigned char>(char const*, char const*, bool, bool, CSphBufferTrait<unsigned char>&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x00000000005dd144 in CSphIndex_VLN::Preread() ()   
#2  0x00007f63518d8e26 in PrereadFunc () at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/taskpreread.cpp:48   
#3  0x00007f63518d1e76 in Action (pArg=0x16e8040) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:246   
#4  PerformTask (pArg=0x16e8040) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:344   
#5  JobTick (pArg=0x16e8040) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:620   
#6  JobLoop (pArg=0x16e8040) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:632   
#7  LazyJobs_c::TheadPoolWorker (pArg=0x16e8040) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:678   
#8  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#9  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>0 0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 104 (Thread 0x7f55428d9700 (LWP 13316)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 103 (Thread 0x7f55427d5700 (LWP 13317)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 102 (Thread 0x7f55426d1700 (LWP 13318)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 101 (Thread 0x7f55425cd700 (LWP 13319)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 100 (Thread 0x7f55424c9700 (LWP 13320)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 99 (Thread 0x7f55423c5700 (LWP 13321)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 98 (Thread 0x7f55422c1700 (LWP 13322)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 97 (Thread 0x7f55421bd700 (LWP 13323)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 96 (Thread 0x7f55420b9700 (LWP 13324)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 95 (Thread 0x7f5541fb5700 (LWP 13325)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 94 (Thread 0x7f5541eb1700 (LWP 13326)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 93 (Thread 0x7f5541dad700 (LWP 13327)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 92 (Thread 0x7f5541ca9700 (LWP 13328)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 91 (Thread 0x7f5541ba5700 (LWP 13329)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 90 (Thread 0x7f5541aa1700 (LWP 13330)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 89 (Thread 0x7f554199d700 (LWP 13331)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 88 (Thread 0x7f5541899700 (LWP 13332)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 87 (Thread 0x7f5541795700 (LWP 13333)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 86 (Thread 0x7f5541691700 (LWP 13334)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 85 (Thread 0x7f554158d700 (LWP 13335)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 84 (Thread 0x7f5541489700 (LWP 13336)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 83 (Thread 0x7f5541385700 (LWP 13337)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 82 (Thread 0x7f5541281700 (LWP 13338)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 81 (Thread 0x7f554117d700 (LWP 13339)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 80 (Thread 0x7f5541079700 (LWP 13340)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 79 (Thread 0x7f5540f75700 (LWP 13341)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 78 (Thread 0x7f5540e71700 (LWP 13342)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 77 (Thread 0x7f5540d6d700 (LWP 13343)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 76 (Thread 0x7f5540c69700 (LWP 13344)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 75 (Thread 0x7f5540b65700 (LWP 13345)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 74 (Thread 0x7f5540a61700 (LWP 13346)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 73 (Thread 0x7f554095d700 (LWP 13347)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 72 (Thread 0x7f5540859700 (LWP 13348)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 71 (Thread 0x7f5540755700 (LWP 13351)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 70 (Thread 0x7f5540651700 (LWP 13352)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 69 (Thread 0x7f554054d700 (LWP 13353)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 68 (Thread 0x7f5540449700 (LWP 13354)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 67 (Thread 0x7f5540345700 (LWP 13355)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 66 (Thread 0x7f5540241700 (LWP 13356)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 65 (Thread 0x7f554013d700 (LWP 13357)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 64 (Thread 0x7f5540039700 (LWP 13358)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 63 (Thread 0x7f553ff35700 (LWP 13359)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 62 (Thread 0x7f553fe31700 (LWP 13360)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 61 (Thread 0x7f553fd2d700 (LWP 13361)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 60 (Thread 0x7f553fc29700 (LWP 13362)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 59 (Thread 0x7f553fb25700 (LWP 13363)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 58 (Thread 0x7f553fa21700 (LWP 13364)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 57 (Thread 0x7f553f91d700 (LWP 13365)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 56 (Thread 0x7f553f819700 (LWP 13366)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 55 (Thread 0x7f553f715700 (LWP 13367)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 54 (Thread 0x7f553f611700 (LWP 13368)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 53 (Thread 0x7f553f50d700 (LWP 13369)):   
#0  0x0000003a37ce1523 in select () from /lib64/libc.so.6   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x00000000005a4514 in sphSleepMsec ()   
#2  0x000000000065da71 in sphBacktrace(int, bool) ()   
#3  0x000000000051fe5e in SphCrashLogger_c::HandleCrash(int) ()   
#4  <signal handler called>   
#5  0x0000003a37c7ba1c in free () from /lib64/libc.so.6   
#6  0x000000000057dd6a in CSphString::SafeFree() ()   
#7  0x000000000058bf4e in CSphQuery::~CSphQuery() ()   
#8  0x000000000059fb0f in CSphinxqlSession::Execute(CSphString const&, ISphOutputBuffer&, unsigned char&, Threads::ThdDesc_t&) ()   
#9  0x000000000057b4e5 in LoopClientMySQL(unsigned char&, CSphinxqlSession&, CSphString&, int, bool, Threads::ThdDesc_t&, InputBuffer_c&, ISphOutputBuffer&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>0 0x000000000057be8e in ThdJobQL_t::Call() ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>1 0x00000000006688a0 in CSphThdPool::Tick(void*) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>2 0x0000000000664235 in sphThreadProcWrapper(void*) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>3 0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>4 0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 52 (Thread 0x7f553f409700 (LWP 13370)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 51 (Thread 0x7f553f305700 (LWP 13371)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 50 (Thread 0x7f553f201700 (LWP 13372)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 49 (Thread 0x7f553f0fd700 (LWP 13373)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 48 (Thread 0x7f553eff9700 (LWP 13374)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 47 (Thread 0x7f553eef5700 (LWP 13375)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 46 (Thread 0x7f553edf1700 (LWP 13376)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 45 (Thread 0x7f553eced700 (LWP 13377)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 44 (Thread 0x7f553ebe9700 (LWP 13378)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 43 (Thread 0x7f553eae5700 (LWP 13379)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 42 (Thread 0x7f553e9e1700 (LWP 13380)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 41 (Thread 0x7f553e8dd700 (LWP 13381)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 40 (Thread 0x7f553e7d9700 (LWP 13382)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 39 (Thread 0x7f553e6d5700 (LWP 13383)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 38 (Thread 0x7f553e5d1700 (LWP 13384)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 37 (Thread 0x7f553e4cd700 (LWP 13385)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 36 (Thread 0x7f553e3c9700 (LWP 13386)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 35 (Thread 0x7f553e2c5700 (LWP 13387)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 34 (Thread 0x7f553e1c1700 (LWP 13388)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 33 (Thread 0x7f553e0bd700 (LWP 13389)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 32 (Thread 0x7f553dfb9700 (LWP 13390)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 31 (Thread 0x7f553deb5700 (LWP 13391)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 30 (Thread 0x7f553ddb1700 (LWP 13392)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 29 (Thread 0x7f553dcad700 (LWP 13393)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 28 (Thread 0x7f553dba9700 (LWP 13394)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 27 (Thread 0x7f553daa5700 (LWP 13395)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 26 (Thread 0x7f553d9a1700 (LWP 13396)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 25 (Thread 0x7f553d89d700 (LWP 13397)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 24 (Thread 0x7f553d799700 (LWP 13398)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 23 (Thread 0x7f553d695700 (LWP 13399)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 22 (Thread 0x7f553d591700 (LWP 13400)):   
#0  0x0000003a37c89cec in _wordcopy_fwd_aligned () from /lib64/libc.so.6   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000003a37c83b22 in memmove () from /lib64/libc.so.6   
#2  0x00000000005e4c2c in DiskIndexQwordSetup_c::Setup(ISphQword*) const ()   
#3  0x00000000007c2647 in CreateQueryWord(XQKeyword_t const&, ISphQwordSetup const&, CSphDict*) ()   
#4  0x00000000007c7c57 in ExtNode_i::Create(XQKeyword_t const&, XQNode_t const*, ISphQwordSetup const&, bool) ()   
#5  0x00000000007c9a9e in ExtNode_i::Create(XQNode_t const*, ISphQwordSetup const&, bool) ()   
#6  0x00000000007c90c6 in ExtNode_i::Create(XQNode_t const*, ISphQwordSetup const&, bool) ()   
#7  0x000000000073e747 in ExtRanker_c::ExtRanker_c(XQQuery_t const&, ISphQwordSetup const&, bool, bool, bool) ()   
#8  0x000000000074fa1b in ExtRanker_State_T<RankerState_Expr_fn<false, true>, true>::ExtRanker_State_T(XQQuery_t const&, ISphQwordSetup const&, bool, bool) ()   
#9  0x0000000000740ede in sphCreateRanker(XQQuery_t const&, CSphQuery const*, CSphQueryResult*, ISphQwordSetup const&, CSphQueryContext const&, ISphSchema const&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>0 0x00000000005ca76c in CSphIndex_VLN::ParsedMultiQuery(CSphQuery const*, CSphQueryResult*, int, ISphMatchSorter**, XQQuery_t const&, CSphDict*, CSphMultiQueryArgs const&, CSphQueryNodeCache*, SphWordStatChecker_t const&) const ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>1 0x00000000005edf40 in CSphIndex_VLN::MultiQuery(CSphQuery const*, CSphQueryResult*, int, ISphMatchSorter**, CSphMultiQueryArgs const&) const ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>2 0x00000000005eea1f in CSphIndex_VLN::MultiQueryEx(int, CSphQuery const*, CSphQueryResult**, ISphMatchSorter**, CSphMultiQueryArgs const&) const ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>3 0x0000000000563ddc in SearchHandler_c::RunLocalSearches() ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>4 0x0000000000575b93 in SearchHandler_c::RunSubset(int, int) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>5 0x0000000000576b15 in SearchHandler_c::RunQueries() ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>6 0x0000000000577168 in HandleMysqlSelect(SqlRowBuffer_c&, SearchHandler_c&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>7 0x00000000005a008f in CSphinxqlSession::Execute(CSphString const&, ISphOutputBuffer&, unsigned char&, Threads::ThdDesc_t&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>8 0x000000000057b4e5 in LoopClientMySQL(unsigned char&, CSphinxqlSession&, CSphString&, int, bool, Threads::ThdDesc_t&, InputBuffer_c&, ISphOutputBuffer&) ()   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>9 0x000000000057be8e in ThdJobQL_t::Call() ()   
#20 0x00000000006688a0 in CSphThdPool::Tick(void*) ()   
#21 0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#22 0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#23 0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 21 (Thread 0x7f553d48d700 (LWP 13401)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 20 (Thread 0x7f553d389700 (LWP 13402)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 19 (Thread 0x7f553d285700 (LWP 13403)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 18 (Thread 0x7f553d181700 (LWP 13404)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 17 (Thread 0x7f553d07d700 (LWP 13405)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 16 (Thread 0x7f553cf79700 (LWP 13406)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 15 (Thread 0x7f553ce75700 (LWP 13407)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 14 (Thread 0x7f553cd71700 (LWP 13408)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 13 (Thread 0x7f553cc6d700 (LWP 13409)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 12 (Thread 0x7f553cb69700 (LWP 13410)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 11 (Thread 0x7f553ca65700 (LWP 13411)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 10 (Thread 0x7f553c961700 (LWP 13412)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 9 (Thread 0x7f553c85d700 (LWP 13413)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 8 (Thread 0x7f553c759700 (LWP 13414)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 7 (Thread 0x7f553c655700 (LWP 13415)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 6 (Thread 0x7f553c551700 (LWP 13416)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 5 (Thread 0x7f553c44d700 (LWP 13417)):   
#0  0x0000003a3800b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664cab in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x000000000066882f in CSphThdPool::Tick(void*) ()   
#3  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#4  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#5  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 4 (Thread 0x7f553c147700 (LWP 13419)):   
#0  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664c36 in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x00007f63518d1c72 in JobTick (pArg=0x20b9240) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:604   
#3  JobLoop (pArg=0x20b9240) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:632   
#4  LazyJobs_c::TheadPoolWorker (pArg=0x20b9240) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:678   
#5  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#6  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#7  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 3 (Thread 0x7f5537fff700 (LWP 13420)):   
#0  0x0000003a37ce90a3 in epoll_wait () from /lib64/libc.so.6   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x00007f63518a324e in EpollEvents_c::Wait (this=0x7f55300008c0, timeoutMs=Unhandled dwarf expression opcode 0xf3   
) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdha.cpp:3998   
#2  0x00000000005877ab in CSphNetLoop::LoopNetPoll() ()   
#3  0x0000000000587f7f in CSphNetLoop::ThdLoop(void*) ()   
#4  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#5  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#6  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 2 (Thread 0x7f5537e07700 (LWP 19393)):   
#0  0x0000003a3800ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x0000000000664c36 in AutoEvent_T<false>::WaitEvent(int) ()   
#2  0x00007f63518d1c72 in JobTick (pArg=0x7f55280a4510) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:604   
#3  JobLoop (pArg=0x7f55280a4510) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:632   
#4  LazyJobs_c::TheadPoolWorker (pArg=0x7f55280a4510) at /root/rpmbuild/BUILD/sphinx-3.2.2/manticore-3.2.2/src/searchdtask.cpp:678   
#5  0x0000000000664235 in sphThreadProcWrapper(void*) ()   
#6  0x0000003a38007aa1 in start_thread () from /lib64/libpthread.so.0   
#7  0x0000003a37ce8aad in clone () from /lib64/libc.so.6   
   
Thread 1 (Thread 0x7f635165b720 (LWP 13152)):   
#0  0x0000003a37ce1523 in select () from /lib64/libc.so.6   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x00000000005a4514 in sphSleepMsec ()   
#2  0x000000000056d3c2 in TickHead() ()   
#3  0x000000000057011a in ServiceMain(int, char**) ()   
#4  0x000000000051d729 in main ()   
   
Main thread:   
#0  0x0000003a37ce1523 in select () from /lib64/libc.so.6   
<a href ="https://Github.com/manticoresoftware/manticoresearch/issues/1">#1</a>  0x00000000005a4514 in sphSleepMsec ()   
#2  0x000000000056d3c2 in TickHead() ()   
#3  0x000000000057011a in ServiceMain(int, char**) ()   
#4  0x000000000051d729 in main ()   
   
Local variables:   
No symbol table info available.   
--- 3 active threads ---   
thd 0, proto sphinxql, state query, command select   
thd 1, proto sphinxql, state query, command select   
thd 2, proto sphinxql, state handshake, command SYSTEM   
------- CRASH DUMP END -------

In case you've faced a crash what indextool --check returns:

indextool --config /usr/local/etc/sphinx.conf --check mdb_morf_archive_idx   
Manticore 3.2.2 afd6046@191226 release   
Copyright (c) 2001-2016, Andrew Aksyonoff   
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)   
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)   
   
using config file '/usr/local/etc/sphinx.conf'...   
checking index 'mdb_morf_archive_idx'...   
checking dictionary...   
checking data...   
checking rows...   
checking attribute blocks index...   
checking kill-list...   
checking dead row map...   
checking doc-id lookup...   
check passed, 33153.9 sec elapsed

Config file:

index mdb_morf_idx : mdb_base_idx   
{   
        preopen = 1   
   
        min_stemming_len = 4   
        min_word_len = 1   
        index_exact_words = 1   
   
        index_sp = 1   
   
        morphology = lemmatize_ru, lemmatize_en, \   
                                libstemmer_es, libstemmer_de, libstemmer_fr, libstemmer_nl, libstemmer_it, \   
                                libstemmer_hu, libstemmer_fi, libstemmer_no, libstemmer_pt, libstemmer_ro, libstemmer_da, \   
                                libstemmer_sv, libstemmer_tr   
        stopwords       = /usr/local/var/data/russian_stopwords.txt /usr/local/var/data/english_stopwords.txt \    
                                /usr/local/var/data/arabic_french_stopwords.txt /usr/local/var/data/dutch_stopwords.txt \   
                                /usr/local/var/data/hebrew_stopwords.txt /usr/local/var/data/portuguese_stopwords.txt \   
                                /usr/local/var/data/swedish_finnish_hungarian_stopwords.txt /usr/local/var/data/arabic_stopwords.txt \   
                                /usr/local/var/data/english_stopwords.txt  /usr/local/var/data/italian_stopwords.txt \   
                                /usr/local/var/data/romanian_stopwords.txt /usr/local/var/data/swedish_stopwords.txt \   
                                /usr/local/var/data/armenian_stopwords.txt /usr/local/var/data/finnish_stopwords.txt \   
                                /usr/local/var/data/japanese_stopwords.txt /usr/local/var/data/russian_stopwords.txt \   
                                /usr/local/var/data/turkish_stopwords.txt /usr/local/var/data/chinese_stopwords.txt \   
                                /usr/local/var/data/french_stopwords.txt  /usr/local/var/data/lithuanian_stopwords.txt \   
                                /usr/local/var/data/serbian_stopwords.txt /usr/local/var/data/ukraine_stopwords.txt \   
                                /usr/local/var/data/czech_stopwords.txt /usr/local/var/data/german_stopwords.txt \   
                                /usr/local/var/data/norwegian_stopwords.txt /usr/local/var/data/slovenian_stopwords.txt \   
                                /usr/local/var/data/danish_stopwords.txt /usr/local/var/data/greek_stopwords.txt \   
                                /usr/local/var/data/polish_stopwords.txt /usr/local/var/data/spanish_stopwords.txt \   
                                /usr/local/var/data/fs_mediadb_stopwords.txt   
   
        stopwords_unstemmed = 1   
}   
   
index mdb_morf_archive_idx : mdb_morf_idx   
{   
        killlist_target = mdb_morf_archive_old_idx:kl   
        source = mdb_morf_archive   
        path = /usr/local/var/data/mdb_morf_archive_idx/mdb_morf_archive_idx   
}
@klirichek
Copy link
Contributor

  1. Did you tried with fresher version? 3.2.2 afd6046@191226 - last numbers means '2019-12-26', and another versions were released also since this one.

  2. In config - there is not enough info, no way to know the schema. If you can't dump it - just describe. What columns, in which order and types returned by source-query when indexing. And which of them are in source config (i.e., becomes attributes, apart the rest becoming ft-fields)

@tomatolog
Copy link
Contributor

from crash log it seems like memory corruption and in case it reproduces with the same query to investigate it further we need you index files (mdb_morf_archive_idx)

@hgfdd
Copy link
Author

hgfdd commented Jul 2, 2020

  1. Did you tried with fresher version? 3.2.2 afd6046@191226 - last numbers means '2019-12-26', and another versions were released also since this one.

I have updated my manticore to 3.4.2 yestarday, but i face the same problem.

  1. In config - there is not enough info, no way to know the schema. If you can't dump it - just describe. What columns, in which order and types returned by source-query when indexing. And which of them are in source config (i.e., becomes attributes, apart the rest becoming ft-fields)

I managed to reproduce the problem on smaller index. I have uploaded index, configs and script to reproduce the problem to ftp.

Waiting for your responce. Thanks in advance!

@githubmanticore
Copy link
Contributor

➤ Stan commented:

with data provided at Debug build it crashed on the 1st run with following stack

/home/stas/bin/searchd(_ZN19RankerState_Expr_fnILb0ELb1EED1Ev+0x4a)[0x87d99a]  
/home/stas/bin/searchd(_ZN16ExtRanker_Expr_TILb0ELb1EED0Ev+0x30)[0x87eac0]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN16ParsedMultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK9XQQuery_tP8CSphDictRK18CSphMultiQueryArgsP18CSphQueryNodeCache+0xbc4)[0x6ba804]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x657)[0x6d29a7]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN12MultiQueryExEiPK9CSphQueryPP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x964)[0x6d3584]  
/home/stas/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x8f9)[0x5d8f59]  
/home/stas/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii+0x18ff)[0x5dbe7f]  
/home/stas/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv+0x95)[0x5dc0c5]  
/home/stas/bin/searchd(_Z17HandleMysqlSelectR11RowBuffer_iR15SearchHandler_c+0x3e8)[0x5dd2a8]  
/home/stas/bin/searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR11RowBuffer_iRN7Threads9ThdDesc_tE+0x1e2f)[0x60be5f]

at RankerState_Expr_fn<NEED_PACKEDFACTORS, HANDLE_DUPES>::Update
line uQposFixed = m_dTermDupes[uQposFixed]; with out of bound m_dTermDupes index. Seems like duplicated query terms issue.

At Release build it crashed after multiple queries passed sometimes with similar stack and sometimes with memory corruption.

@hgfdd
Copy link
Author

hgfdd commented Jul 6, 2020

➤ Stan commented:

with data provided at Debug build it crashed on the 1st run with following stack

/home/stas/bin/searchd(_ZN19RankerState_Expr_fnILb0ELb1EED1Ev+0x4a)[0x87d99a]  
/home/stas/bin/searchd(_ZN16ExtRanker_Expr_TILb0ELb1EED0Ev+0x30)[0x87eac0]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN16ParsedMultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK9XQQuery_tP8CSphDictRK18CSphMultiQueryArgsP18CSphQueryNodeCache+0xbc4)[0x6ba804]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN10MultiQueryEPK9CSphQueryP15CSphQueryResultiPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x657)[0x6d29a7]  
/home/stas/bin/searchd(_ZNK13CSphIndex_VLN12MultiQueryExEiPK9CSphQueryPP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x964)[0x6d3584]  
/home/stas/bin/searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x8f9)[0x5d8f59]  
/home/stas/bin/searchd(_ZN15SearchHandler_c9RunSubsetEii+0x18ff)[0x5dbe7f]  
/home/stas/bin/searchd(_ZN15SearchHandler_c10RunQueriesEv+0x95)[0x5dc0c5]  
/home/stas/bin/searchd(_Z17HandleMysqlSelectR11RowBuffer_iR15SearchHandler_c+0x3e8)[0x5dd2a8]  
/home/stas/bin/searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR11RowBuffer_iRN7Threads9ThdDesc_tE+0x1e2f)[0x60be5f]

at RankerState_Expr_fn<NEED_PACKEDFACTORS, HANDLE_DUPES>::Update
line uQposFixed = m_dTermDupes[uQposFixed]; with out of bound m_dTermDupes index. Seems like duplicated query terms issue.

At Release build it crashed after multiple queries passed sometimes with similar stack and sometimes with memory corruption.

This bug is kind of critical from me. Clients form a lot equal queries and a lot of questions arise. How long will it take to fix this bug or maybe you can send some temporary patch with fix?

@sanikolaev
Copy link
Collaborator

@hgfdd

Unfortunately this bug can't be fixed in 5 minutes, i.e. it's not just some mistype which is causing the issue and most likely will take few hours or more than a day to localize the root cause. So we cannot guarantee it will be fixed this week. If it's mission critical to have it fixed urgently please consider our commercial support. Contact us at [email protected] and we can have a call and discuss what we can do.

@hgfdd
Copy link
Author

hgfdd commented Aug 4, 2020

@hgfdd

Unfortunately this bug can't be fixed in 5 minutes, i.e. it's not just some mistype which is causing the issue and most likely will take few hours or more than a day to localize the root cause. So we cannot guarantee it will be fixed this week. If it's mission critical to have it fixed urgently please consider our commercial support. Contact us at [email protected] and we can have a call and discuss what we can do.

Hello!

How is it going? In what version you are planning to fix this bug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@tomatolog @klirichek @hgfdd @sanikolaev @githubmanticore