With both ngram and morphology = icu_chinese enabled the indexer crashes

[rlattuad]

Manticore 3.1.0 445e806@190716 release
Copyright © 2001-2016, Andrew Aksyonoff
Copyright © 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright © 2017-2019, Manticore Software LTD (http://manticoresearch.com)

using config file ‘/etc/sphinx/sphinx.conf’…
indexing index ‘myHB_index2’…
*** Oops, indexer crashed! Please send the following report to developers.
Manticore 3.1.0 445e806@190716 release
-------------- report begins here ---------------
Current document: docid=85, hits=2783
Current batch: minid=0, maxid=0
Hit pool start: docid=0, hit=0
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DICU_IS_SHARED=1 -DDL_ICU=1 -DICU_LIB=libicuuc.so.50 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=etc/sphinx
Host OS is Linux runner-fa6cab46-project-3858465-concurrent-0 4.19.23-coreos-r1 #1 SMP Mon Feb 25 23:40:01 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x21b0010, thread stack size = 0x100000
Trying manual backtrace:
Stack looks OK, attempting backtrace.
Trying system backtrace:
begin of system symbols:
/usr/bin/indexer(_Z12sphBacktraceib 0x90)[0x5c9220]
/usr/bin/indexer(_Z7sigsegvi 0xa0)[0x50bfc0]
/lib64/libpthread.so.0( 0xf5d0)[0x7f67cd14d5d0]
/lib64/libc.so.6(gsignal 0x37)[0x7f67cbf6a207]
/lib64/libc.so.6(abort 0x148)[0x7f67cbf6b8f8]
/lib64/libc.so.6( 0x78d27)[0x7f67cbfacd27]
/lib64/libc.so.6( 0x81489)[0x7f67cbfb5489]
/usr/bin/indexer(_ZN16FieldFilterICU_c5ApplyEPKhiRN3sph8Vector_TIhNS2_13DefaultCopy_TIhEENS2_14DefaultRelimitENS2_16DefaultStorage_TIhEEEEb 0xbf)[0x748daf]
/usr/bin/indexer(_ZN19CSphSource_Document15IterateDocumentERbR10CSphString 0x270)[0x554fd0]
/usr/bin/indexer(_ZN13CSphIndex_VLN5BuildERKN3sph8Vector_TIP10CSphSourceNS0_13DefaultCopy_TIS3_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS3_EEEEii 0xdf3)[0x56bd73]
/usr/bin/indexer(_Z7DoIndexRK17CSphConfigSectionPKcRK15CSphOrderedHashIS_10CSphString15CSphStrHashFuncLi256EEbP8_IO_FILE 0x119d)[0x51280d]
/usr/bin/indexer(main 0x19f7)[0x50b307]
/lib64/libc.so.6(__libc_start_main 0xf5)[0x7f67cbf563d5]
/usr/bin/indexer[0x50be54]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter ‘Reporting bugs’ in the documentation
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)
Will run gdb on /usr/bin/indexer, pid 30338

[githubmanticore]

➤ Ilya Kuznetsov commented:

I need your config and a minimum subset of your data to reproduce the issue.

You can upload your data to our write-only FTP for customer data:

ftp: dev.manticoresearch.com
user: manticorebugs
pass: shithappens

[rlattuad]

Files uploaded in github-issue-315

[githubmanticore]

➤ Ilya Kuznetsov commented:

I can't reproduce the issue with the provided data:

Manticore 3.4.3 8dbafc8e@200519 dev  
Copyright (c) 2001-2016, Andrew Aksyonoff  
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)  
Copyright (c) 2017-2020, Manticore Software LTD (http://manticoresearch.com)  
  
using config file 'github-issue-315\manticore.conf'...  
WARNING: key 'read_timeout' is deprecated in github-issue-315\manticore.conf line 690; use 'network_timeout' instead.  
indexing index 'index1_template'...  
ERROR: index 'index1_template': key 'path' not found.  
indexing index 'index2'...  
collected 20 docs, 0.0 MB  
creating lookup: 0.0 Kdocs, 100.0% done  
creating histograms: 0.0 Kdocs, 100.0% done  
sorted 0.0 Mhits, 100.0% done  
total 20 docs, 2343 bytes  
total 1.785 sec, 1312 bytes/sec, 11.20 docs/sec  
total 4 reads, 0.000 sec, 8.7 kb/call avg, 0.0 msec/call avg  
total 15 writes, 0.000 sec, 1.7 kb/call avg, 0.0 msec/call avg  

[githubmanticore]

➤ Ilya Kuznetsov commented:

Why are you using both ngram chars and icu at the same time? What is the point in having both ways of segmentation enabled?

[rlattuad]

Hi, I just had both enabled by mistake, this cause the error and indexer crash, I have then removed icu and everything is fine. Still I filed a crash report as with both settings my indexer crashed. Hope this helps Roberto On Thursday, 21 May 2020, 10:29:59 CEST, Manticore Team <[email protected]> wrote: ➤ Ilya Kuznetsov commented: Why are you using both ngram chars and icu at the same time? What is the point in having both ways of segmentation enabled? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[githubmanticore]

➤ Ilya Kuznetsov commented:

Can't reproduce on local or provided data.