Manticoresearch crashes when count distinct and group by is used

[jgould22]

Describe the bug
Manticore Search crashes while executing the below query.

( It should be noted I encountered this crash while trying to debug another issue where using count distinct with group by is returning correct results for only the first row and every subsequent row is just 1 )

To Reproduce
A docker compose environment as well as sample data is included attached to this issue

Untar the attached tar which contains 4 files
manticore_crash_env.tar.gz

• docker-compose.yml
• manticore.conf
• sample_data.sql
• setup_env.bash

The setup_env.bash will start the compose env and run the query, you may also just inspect it and run the command manually

Steps to reproduce the behavior:
(only a subset of these may be required to reproduce the behaviour but this a stripped down version of my existing workflow that seems to trigger it reliably)

1. Create a mysql instance
2. Load sample data (~30 million rows)
3. Create a manticore instance
4. Configure manticore using manticore conf to pull in 12 plain tables, and 12 real time tables that are connected by a distributed table
5. Load the 12 plain tables from mysql using the indexer
6. Reload the 12 tables and attach them to the real time tables
7. Query the distributed table with the following query SELECT id, COUNT(DISTINCT some_data) from idx_table_rt GROUP BY id;

The query SELECT fkey, COUNT(DISTINCT some_data) from idx_table_rt GROUP BY fkey; does not seem to cause the same crash.

This query however does
SELECT fkey AS fkey, count(DISTINCT some_data) AS some_data_distinct_count FROM idx_table_rt GROUP BY fkey ORDER BY some_data_distinct_count DESC OPTION max_matches=1000, distinct_precision_threshold=0;

Expected behavior
I expect manticore to return the result of the query, and if the crash is related to a configuration error or resource error emit an error but not crash and drop the connection.

Describe the environment:

• Docker env described in attached docker compose file
• Manticoresearch running in docker with version (6.2.12 or 6.2.0)
• Manticore configured as described in the attached tar
• The docker host systems are also as follows (problem occurs on both)
  • Linux tpx 6.4.10-arch1-1 Should I use manticore instead of sphinxsearch? #1 SMP PREEMPT_DYNAMIC Fri, 11 Aug 2023 11:03:36 +0000 x86_64 GNU/Linux
    • 32gb ram
    • 13th Gen Intel© Core™ i7-1365U × 10
    • 1tb ssd
  • Linux manticore 5.15.0-76-generic daemon crash on GROUP N BY #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
    • 64gb ram
    • Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
    • 2tb ssd

Messages from log files:

Docker emits the following log from manticore when this occurs

Crash!!! Handling signal 11

and the process is then rebooted by docker

The client returns

ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

Additional context
indextool --check returns ok for each of the rt tables but prints the following warning

WARNING: failed to load RAM chunks, checking only 1 disk chunks

[PavelShilin89]

Don't forget to pre-download the archive with the configuration files and the mysql dump - archive.zip

MRE

1. Download and install MySQL if it's not already installed. Connect to MySQL using the command: mysql -uroot -p{{your password}}. Don't forget to replace {{your password}} with your password.
2. Create a new database named manticore_test with the SQL command: CREATE DATABASE manticore_test;
3. Import data from a MySQL dump file into the manticore_test database using the command: mysql -uroot -p{{your password}} manticore_test < sample_data.sql
4. Create an idx folder - mkdir idx
5. Run the indexer for your Manticore configuration file, rotating indexes idx_table_0 to idx_table_1 using the command: indexer -c manticore.conf idx_table_{0..1}

6. Start the Manticore search daemon process with your configuration file using the command: searchd --config ./manticore.conf
7. Reload tables - mysql -h127.0.0.1 -P9306 -e "RELOAD TABLES;"

8. Use a loop to attach tables idx_table_$i to idx_table_rt_$i using the ATTACH command in MySQL. Replace $i with the index values from 0 to 1 using the following loop:
for i in {0..1}; do mysql -h0 -P9306 -e "ATTACH TABLE idx_table_$i TO TABLE idx_table_rt_$i WITH TRUNCATE;"; done
9. Run the command : mysql -h0 -P9306 -e "SELECT id, COUNT(DISTINCT some_data) from idx_table_rt GROUP BY id;" and get the error listed below.

Error:

ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

Logs:

------- FATAL: CRASH DUMP -------
[Wed Sep 13 15:01:35.227 2023] [10646]

--- crashed SphinxQL request dump ---
SELECT id, COUNT(DISTINCT some_data) from idx_table_rt GROUP BY id
--- request dump end ---
--- local index:idx_table_rt_0
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.7
Configured with flags: Configured with these definitions: -DDISTR_BUILD=macos -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=/opt/homebrew/opt/zstd/lib/libzstd.1.dylib -DWITH_CURL=1 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=/opt/homebrew/opt/unixodbc/lib/libodbc.2.dylib -DWITH_EXPAT=1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=/opt/homebrew/opt/mysql-client/lib/libmysqlclient.21.dylib -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=/opt/homebrew/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib -DLOCALDATADIR=/opt/homebrew/var/manticore -DFULL_SHARE_DIR=/opt/homebrew/share/manticore
Built on Linux x86_64 for Darwin arm64 (macos)
Stack bottom = 0x130058000, thread stack size = 0x20000
Trying manual backtrace:
Frame pointer is null, manual backtrace failed (did you build with -fomit-frame-pointer?)
Trying system backtrace:
begin of system symbols:
Trying boost backtrace:
 0# sphBacktrace(int, bool) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
 1# CrashLogger::HandleCrash(int) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
 2# _sigtramp in /usr/lib/system/libsystem_platform.dylib
 3# pthread_kill in /usr/lib/system/libsystem_pthread.dylib
 4# abort in /usr/lib/system/libsystem_c.dylib
 5# malloc_vreport in /usr/lib/system/libsystem_malloc.dylib
 6# malloc_report in /usr/lib/system/libsystem_malloc.dylib
 7# find_zone_and_free in /usr/lib/system/libsystem_malloc.dylib
 8# CSphSchemaHelper::FreeDataPtrs(CSphMatch&) const in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
 9# CSphKBufferGroupSorter<MatchGeneric1_fn, UniqHLL_c, 1, false, false>::MoveTo(ISphMatchSorter*, bool) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
10# GlobalSorters_c::MergeResults(VecTraits_T<AggrResult_t>&) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
11# SearchHandler_c::RunLocalSearches() in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
12# SearchHandler_c::RunSubset(int, int) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
13# SearchHandler_c::RunQueries() in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
14# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
15# ClientSession_c::Execute(std::__1::pair<char const*, int>, RowBuffer_i&) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
16# ProcessSqlQueryBuddy(std::__1::pair<char const*, int>, unsigned char&, GenericOutputBuffer_c&) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
17# SqlServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
18# MultiServe(std::__1::unique_ptr<AsyncNetBuffer_c, std::__1::default_delete<AsyncNetBuffer_c>>, std::__1::pair<int, unsigned char>, Proto_e) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
19# std::__1::__function::__func<NetActionAccept_c::Impl_c::ProcessAccept()::$_3, std::__1::allocator<NetActionAccept_c::Impl_c::ProcessAccept()::$_3>, void ()>::operator()() in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd
20# Threads::CoRoutine_c::CreateContext(std::__1::function<void ()>, std::__1::pair<boost::context::stack_context, Threads::StackFlavour_E>)::'lambda'(boost::context::detail::transfer_t)::__invoke(boost::context::detail::transfer_t) in /opt/homebrew/Cellar/manticoresearch/6.2.12-230822-dc5144d35/bin/searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_6), proto mysql, state query, command select
--- Totally 2 threads, and 
1 client-working threads ---
------- CRASH DUMP END -------

[tomatolog]

should be fixed at 490f204