Indexer Crashes #1480
Comments
|
Hi. Is there a way to reproduce it? |
sanikolaev
added
bug
waiting
|
the crash is on loading the config could you provide your config to reproduce this crash locally? |
|
here is config: data source definition############################################################################# source main_src_foo ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_doo_business_search } ############################################################################# index definition############################################################################# table idx_doo_business_search } ############################################################################# index definition############################################################################# table idx_moo_business_search } SOURCE DEFINITION - poo############################################################################# source src_poo_search_full } source src_poo_search_full_test:src_poo_search_full ############################################################################# index definition############################################################################# table idx_poo_search_full wordforms = /etc/manticoresearch/wordforms.txt index idx_poo_search_full_test:idx_poo_search_full ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_search_full } #source src_foo_search_full_test:src_foo_search_full tsvpipe_command = zcat /db/manticore/foo_dedupe/foo_dedupe.gz#} ############################################################################# index definition############################################################################# index idx_foo_search_full #index idx_foo_search_full_test:idx_foo_search_full source = src_foo_search_full_testpath = /db/manticore/foo_dedupe/idx_foo_search_full#} ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_seo } index idx_foo_seo ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_seo_city } index idx_foo_seo_city ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_seo_first } table idx_foo_seo_first ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_seo_state } table idx_foo_seo_state ############################################################################# SOURCE DEFINITION - using main_src_[table_name]############################################################################# source src_foo_test_seo } table idx_foo_test_seo ############################################################################# indexer settings############################################################################# indexer ############################################################################# searchd settings############################################################################# searchd access_plain_attrs = mmap_preread threads = 40mysql_version_string = 5.7.1 index idx_foo_search_full_20230908:idx_foo_search_full index foo_seo_city_20230908:foo_seo_city index foo_seo_first_20230908:foo_seo_first index foo_seo_20230908:foo_seo index foo_seo_opts_20230908:foo_seo_opts index foo_seo_state_20230908:foo_seo_state |
|
So basically I didn't do anything new... went to rotate the index like normal.. ( it SEEMS to be because of the dynamic onfig.. but that's been working too.. so maybe a red herring) |
|
could you attach your config as a file to make sure github parser does not break content trying to format it? |
|
The escaped config is: |
|
Thank you so much for the formatting. I didn't notice the update until now. I'm hoping you guys are closer to figuring it out than I am... |
|
I've tried tons of different things. I've cleaned out every drop of manticore and added it back.. etc.. It SEEMS like the issue only happens when the scripts are being created dynamically ( although I think that may be a red herring ). Anywho... I finally got one to work by only having one index ( no segmentation faults on checkconfig/start) but it never starts. It tries to start forEVER. I left it to 'start' ( it was stuck in reallocation ) and it never finished. Is there a better way to debug or logs I'm missing. As soon as this started the serachd.log was wiped and nothing has been going there since. |
Do you mean indexer never starts? |
This is really strange as searchd doesn't do that. Indexer doesn't know about the searchd log at all. It could be some logrotated which rotated the log. |
|
I can't reproduce the crash if I run indexer like this: Can you please attach the proper config file which has "index foo_seo_city" declaration. Best of all if you can provide a minified version which still reproduces the crash. FYI the very minimal version can look so: |
|
This is the one that works and 'preallocates' for far too long. But honestly, I'd like to be able to know what caused the issue with the segmentation fault. if it's our setup size i'd be so happy to be able to find/show that so we can ask for more space if we need it. lol `source src_foo { } ############################################################################# index definition############################################################################# index idx_foo } indexer Searchd settingssearchd }` |
|
there is a ' right after source on that.. but that was from when I was trying send it to you guys via code type |
|
The config you've attached with the fix of "' right after source", so it looks so: works so: Please provide the following files: If they contain sensitive data, feel free to use our write-only S3 storage - https://manual.manticoresearch.com/Reporting_bugs#Uploading-your-data |
using you config with error symbol I see the error message and correct indexer exit there is no crash |
|
maybe crash related to OS you run indexer - could you try to recreate it in the docker container we provided for release or dev versions? |
|
So I've rewrote everything and I'm pretty sure it's coming from the fact that we upgraded without purging. But I did re-write a ton of stuff too... If I figure out something else, or get the error again, I'll post the info here. Just in case there's some weird bug it might help find. Thanks for all your help. |
Describe the bug
Just tried to run indexer.. got the error message at the bottom.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Describe the environment:
bin/searchd -vorbin/indexer -v): e.g.3.5.0 1d34c49@200722 releaseuname -aif on a Unix-like system):Messages from log files:
Messages from searchd.log and query.log (if applicable).
Additional context
*** Oops, indexer crashed! Please send the following report to developers.
Manticore 6.2.12 dc5144d@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
-------------- report begins here ---------------
Current document: docid=0, hits=0
Current batch: minid=0, maxid=0
Hit pool start: docid=0, hit=0
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=bionic -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (bionic) (cross-compiled)
Stack bottom = 0x0, thread stack size = 0x20000
Trying system backtrace:
begin of system symbols:
indexer(_Z12sphBacktraceib+0x22a)[0x55ad9abd5d0a]
indexer(_Z7sigsegvi+0xbb)[0x55ad9aad11bb]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fcabb896980]
/lib/x86_64-linux-gnu/libc.so.6(+0xb1306)[0x7fcabb544306]
indexer(_ZN3sph10vSprintf_TI15StringBuilder_cEEvPT_PKcP13__va_list_tag+0x914)[0x55ad9abdc434]
indexer(_ZN15StringBuilder_c8vSprintfEPKcP13__va_list_tag+0xee)[0x55ad9aada79e]
indexer(_ZN6TlsMsg3ErrEPKcz+0xb0)[0x55ad9abd0890]
indexer(_ZN16CSphConfigParser5ParseEv+0x1046)[0x55ad9abd2e76]
indexer(_Z11ParseConfigP15CSphOrderedHashIS_I17CSphConfigSection10CSphString15CSphStrHashFuncLi256EES1_S2_Li256EES1_RK11VecTraits_TIcE+0x6a)[0x55ad9abd34ea]
indexer(+0xda6866)[0x55ad9abd3866]
indexer(Z13sphLoadConfigRK10CSphStringbRS+0x9)[0x55ad9abd3749]
indexer(main+0xed7)[0x55ad9aad2787]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fcabb4b4c87]
indexer(_start+0x2a)[0x55ad9aac5b0a]
Trying boost backtrace:
0# sphBacktrace(int, bool) in indexer
1# sigsegv(int) in indexer
2# 0x00007FCABB896980 in /lib/x86_64-linux-gnu/libpthread.so.0
3# 0x00007FCABB544306 in /lib/x86_64-linux-gnu/libc.so.6
4# void sph::vSprintf_T<StringBuilder_c>(StringBuilder_c*, char const*, __va_list_tag*) in indexer
5# StringBuilder_c::vSprintf(char const*, __va_list_tag*) in indexer
6# TlsMsg::Err(char const*, ...) in indexer
7# CSphConfigParser::Parse() in indexer
8# ParseConfig(CSphOrderedHash<CSphOrderedHash<CSphConfigSection, CSphString, CSphStrHashFunc, 256>, CSphString, CSphStrHashFunc, 256>*, CSphString, VecTraits_T const&) in indexer
9# 0x000055AD9ABD3866 in indexer
10# sphLoadConfig(CSphString const&, bool, CSphString&) in indexer
11# main in indexer
12# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
13# _start in indexer
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Will run gdb on '/usr/bin/indexer', pid '31523'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fcabb8962f4 in __waitpid (pid=31568, stat_loc=0x7ffcea0d7c54, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id Target Id Frame
Thread 1 (Thread 0x7fcabcc1efc0 (LWP 31523)):
#0 0x00007fcabb8962f4 in __waitpid (pid=31568, stat_loc=0x7ffcea0d7c54, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x000055ad9abd55de in sphDumpGdb(int, char const*, char const*) ()
#2 0x000055ad9abd5ea1 in sphBacktrace(int, bool) ()
#3 0x000055ad9aad11bb in sigsegv(int) ()
#4
#5 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#6 0x000055ad9abdc434 in void sph::vSprintf_T<StringBuilder_c>(StringBuilder_c*, char const*, __va_list_tag*) ()
#7 0x000055ad9aada79e in StringBuilder_c::vSprintf(char const*, __va_list_tag*) ()
#8 0x000055ad9abd0890 in TlsMsg::Err(char const*, ...) ()
#9 0x000055ad9abd2e76 in CSphConfigParser::Parse() ()
#10 0x000055ad9abd34ea in ParseConfig(CSphOrderedHash<CSphOrderedHash<CSphConfigSection, CSphString, CSphStrHashFunc, 256>, CSphString, CSphStrHashFunc, 256>*, CSphString, VecTraits_T const&) ()
#11 0x000055ad9abd3866 in ?? ()
#12 0x000055ad9abd3749 in sphLoadConfig(CSphString const&, bool, CSphString&) ()
#13 0x000055ad9aad2787 in main ()
Main thread:
#0 0x00007fcabb8962f4 in __waitpid (pid=31568, stat_loc=0x7ffcea0d7c54, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x000055ad9abd55de in sphDumpGdb(int, char const*, char const*) ()
#2 0x000055ad9abd5ea1 in sphBacktrace(int, bool) ()
#3 0x000055ad9aad11bb in sigsegv(int) ()
#4
#5 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#6 0x000055ad9abdc434 in void sph::vSprintf_T<StringBuilder_c>(StringBuilder_c*, char const*, __va_list_tag*) ()
#7 0x000055ad9aada79e in StringBuilder_c::vSprintf(char const*, __va_list_tag*) ()
#8 0x000055ad9abd0890 in TlsMsg::Err(char const*, ...) ()
#9 0x000055ad9abd2e76 in CSphConfigParser::Parse() ()
#10 0x000055ad9abd34ea in ParseConfig(CSphOrderedHash<CSphOrderedHash<CSphConfigSection, CSphString, CSphStrHashFunc, 256>, CSphString, CSphStrHashFunc, 256>*, CSphString, VecTraits_T const&) ()
#11 0x000055ad9abd3866 in ?? ()
#12 0x000055ad9abd3749 in sphLoadConfig(CSphString const&, bool, CSphString&) ()
#13 0x000055ad9aad2787 in main ()
Local variables:
resultvar = 18446744073709551104
sc_ret =
The text was updated successfully, but these errors were encountered: