Crash on replace

[rodion-k]

Describe the environment

Manticore Search version (top line in output of bin/searchd -v or bin/indexer -v): Manticore 3.4.0 b212975@200327 release

OS version (uname -a if on a Unix-like system): Linux localhost.localdomain 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build version:

Describe the problem:
Searchd crashed

Steps to reproduce:

Attributes part_payment_local_cities and part_payment_delivery_cities have been added via alter table just before crash.
Executed query:
REPLACE INTO vproducts (id, product_id, param_name, delivery_city_id, region_id, filter, category_id, category_id_field, custom_model, producer_id, p1, p2, category_name, category_name_field, category_code, section_name, section_code, producer_name, part_payment_local_cities, part_payment_delivery_cities) VALUES (1264190, 1264190, 'Aral SuperTurboral LA 5W-30 20 л', (2,3), (4), (), 3301, '3301', 'SuperTurboral LA', 3204, 3000, 3300, 'Масла', 'Масла', 100, 'Авто', 550, 'Aral', (), ())

Messsages from log files:
------- FATAL: CRASH DUMP -------
[Tue May 5 11:53:44.107 2020] [ 6653]

--- crashed SphinxQL request dump ---
REPLACE INTO vproducts (id, product_id, param_name, delivery_city_id, region_id, filter,
category_id, category_id_field, custom_model, producer_id, p1, p2, category_name,
category_name_field, category_code, section_name, section_code, producer_name, part_payment_local_cities,
part_payment_delivery_cities) VALUES (1264190, 1264190, 'Aral SuperTurboral LA 5W-30
20 л', (2,3), (4), (), 3301, '3301', 'SuperTurboral LA', 3204, 3000, 3300, 'Масла',
'Масла', 100, 'Авто', 550, 'Aral', (), ())
--- request dump end ---
--- local index:
Manticore 3.4.0 b212975@200327 release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=rhel7 -DUSE_SSL=ON -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_ICU=1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1 -DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SOVERSION=31 -DSYSCONFDIR=/etc/manticoresearch
Host OS is Linux runner-ed2dce3a-project-3858465-concurrent-0 4.19.78-coreos #1 SMP Mon Oct 14 22:56:39 -00 2019 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f74aa116d7f, thread stack size = 0x100000
Trying manual backtrace:
Frame pointer is null, manual backtrace failed (did you build with -fomit-frame-pointer?)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib 0x90)[0x710e20]
/usr/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi 0x1fe)[0x58409e]
/lib64/libpthread.so.0( 0xf5f0)[0x7f79181325f0]
/usr/bin/searchd(_ZN9RtAccum_t11AddDocumentEP8ISphHitsRK9CSphMatchbiPPKcRK11VecTraits_TIlEPKN17DocstoreBuilder_i5Doc_tE 0x1ea)[0x856b9a]
/usr/bin/searchd(_ZN9RtIndex_c11AddDocumentEP8ISphHitsRK9CSphMatchbPPKcRK11VecTraits_TIlEPKN17DocstoreBuilder_i5Doc_tER10CSphStringSH_P9RtAccum_t 0x66)[0x8571f6]
/usr/bin/searchd(_ZN9RtIndex_c11AddDocumentERK11VecTraits_TIS0_IKcEER9CSphMatchbRK10CSphStringPPS1_RKS0_IlERS8_SG_P9RtAccum_t 0x59c)[0x8577ac]
/usr/bin/searchd(_Z20sphHandleMysqlInsertR19StmtErrorReporter_iR9SqlStmt_tbbR10CSphStringR16CSphSessionAccum13ESphCollationRN3sph8Vector_TIlNS8_13DefaultCopy_TIlEENS8_14DefaultRelimitENS8_16DefaultStorage_TIlEEEE 0x185e)[0x5c489e]
/usr/bin/searchd(_ZN16CSphinxqlSession7ExecuteERK10CSphStringR11RowBuffer_iRN7Threads9ThdDesc_tE 0x17ff)[0x5f49af]
/usr/bin/searchd(_Z15LoopClientMySQLRhR16CSphinxqlSessionR10CSphStringibRN7Threads9ThdDesc_tER13InputBuffer_cR16ISphOutputBuffer 0x322)[0x5d1022]
/usr/bin/searchd(_ZN10ThdJobQL_t4CallEv 0x25f)[0x65af8f]
/usr/bin/searchd(_ZN11CSphThdPool4TickEPv 0xa0)[0x719240]
/usr/bin/searchd(_Z20sphThreadProcWrapperPv 0x23)[0x7149a3]
/lib64/libpthread.so.0( 0x7e65)[0x7f791812ae65]
/lib64/libc.so.6(clone 0x6d)[0x7f791694188d]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)
Dump with GDB via watchdog
[Tue May 5 11:53:44.426 2020] [23865] watchdog: got USR1, performing dump of child's stack
Will run gdb on '/usr/bin/searchd', pid '6653'
--- 1 active threads ---
thd 0, proto sphinxql, state query, command replace
------- CRASH DUMP END -------

In case you've faced a crash what indextool --check returns:
indextool_result.txt

describe vproducts:

 ------------------------------ ------------ ------------     
| Field                        | Type       | Properties |    
 ------------------------------ ------------ ------------     
| id                           | bigint     |            |    
| param_name_field             | field      | indexed    |    
| category_name_field          | field      | indexed    |    
| category_id_field            | field      | indexed    |    
| custom_model                 | field      | indexed    |    
| filter_captions              | field      | indexed    |    
| product_id                   | uint       |            |    
| uk_param_name                | string     |            |    
| param_name                   | string     |            |    
| filter                       | mva        |            |    
| category_id                  | uint       |            |    
| category_code                | string     |            |    
| p1                           | uint       |            |    
| p2                           | uint       |            |    
| category_name                | string     |            |    
| section_code                 | string     |            |    
| section_name                 | string     |            |    
| producer_id                  | uint       |            |    
| producer_name                | string     |            |    
| json                         | json       |            |    
| delivery_json                | json       |            |    
| region_id                    | mva        |            |    
| delivery_city_id             | mva        |            |    
| popularity                   | uint       |            |    
| discount                     | json       |            |    
| discount_delivery            | json       |            |    
| param_name_field_len         | tokencount |            |    
| category_name_field_len      | tokencount |            |    
| category_id_field_len        | tokencount |            |    
| custom_model_len             | tokencount |            |    
| filter_captions_len          | tokencount |            |    
| part_payment_local_cities    | mva        |            |    
| part_payment_delivery_cities | mva        |            |    
 ------------------------------ ------------ ------------     

mysql> show index vproducts settings\G

*************************** 1. row ***************************    
Variable_name: settings    
        Value: min_prefix_len = 2    
index_exact_words = 1    
index_field_lengths = 1    
charset_table = 0..9, A..Z->a..z, a..z, U 410..U 42F->U 430..U 44F, U 430..U 44F, U 404->U 454, U 454, U 406->U 456, U 456, U 407->U 457, U 457, U 490->U 491, U 491, U 401->U 435, U 451->U 435    
blend_chars = .    
morphology = stem_enru    
min_stemming_len = 4    
stopwords = /etc/manticoresearch/stopwords.txt    

[githubmanticore]

fixed #1338 crash after alter add column to index with index_field_lengths; fixed github#335; added regressions to test 378; fixed model at test 214 via commit 9d3395a

[tomatolog]

just fixed the crash at 9d3395a and will be published next release.

However index with index_exact_words after ALTER ... ADD ... got broken. You need to delete such index and recreate it from scratch and repopulate it with data.