Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drop column crush #1101

Closed
FunkForever opened this issue Apr 20, 2023 · 5 comments
Closed

Drop column crush #1101

FunkForever opened this issue Apr 20, 2023 · 5 comments
Labels
bug waiting Waiting for the original poster (in most cases) or something else

Comments

@FunkForever
Copy link
Contributor

FunkForever commented Apr 20, 2023 •
edited
Loading

Describe the bug
When executing a request ALTER TABLE pupils DROP COLUMN statusesinorg, the Manticore service crashes

Describe the environment:

  • Manticore 6.0.4 1a3a4ea@230314
  • CentOS Linux release 7.9.2009 (Core)

Messages from log files:
------- FATAL: CRASH DUMP -------
[Thu Apr 20 12:46:52.740 2023] [ 2680]

--- crashed SphinxQL request dump ---
ALTER TABLE pupils DROP COLUMN statusesinorg
--- request dump end ---
--- local index:�r]#�
Manticore 6.0.4 1a3a4ea@230314 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with Clang 15.0.4
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel7 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel7) (cross-compiled)
Stack bottom = 0x7fe88406a4b0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7fe884070000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd(_Z12sphBacktraceib+0x22a)[0x55ac6539305a]
/usr/bin/searchd(_ZN11CrashLogger11HandleCrashEi+0x355)[0x55ac652565c5]
/lib64/libpthread.so.0(+0xf630)[0x7fea2e52f630]
/usr/bin/searchd(+0xd3a7e6)[0x55ac6578f7e6]
/usr/bin/searchd(_ZN18IndexAlterHelper_c26Alter_AddRemoveRowwiseAttrERK10CSphSchemaS2_PKjjPKhR14WriteWrapper_cS8_bRK10CSphString+0x57c)[0x55ac6578dccc]
/usr/bin/searchd(_ZN13CSphIndex_VLN18AddRemoveAttributeEbRK18AttrAddRemoveCtx_tR10CSphString+0x664)[0x55ac652e0c94]
/usr/bin/searchd(_ZN9RtIndex_c18AddRemoveAttributeEbRK18AttrAddRemoveCtx_tR10CSphString+0x3d5)[0x55ac656516b5]
/usr/bin/searchd(+0x84d16e)[0x55ac652a216e]
/usr/bin/searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i+0x190e)[0x55ac6529ec7e]
/usr/bin/searchd(_Z20ProcessSqlQueryBuddySt4pairIPKciERhR16ISphOutputBuffer+0x47)[0x55ac651fd677]
/usr/bin/searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x10b8)[0x55ac651e9b28]
/usr/bin/searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x43)[0x55ac651e59f3]
/usr/bin/searchd(+0x791584)[0x55ac651e6584]
/usr/bin/searchd(ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9+0x1c)[0x55ac65b5971c]
/usr/bin/searchd(make_fcontext+0x2f)[0x55ac65b7856f]
Trying boost backtrace:
0# sphBacktrace(int, bool) in /usr/bin/searchd
1# CrashLogger::HandleCrash(int) in /usr/bin/searchd
2# 0x00007FEA2E52F630 in /lib64/libpthread.so.0
3# 0x000055AC6578F7E6 in /usr/bin/searchd
4# IndexAlterHelper_c::Alter_AddRemoveRowwiseAttr(CSphSchema const&, CSphSchema const&, unsigned int const*, unsigned int, unsigned char const*, WriteWrapper_c&, WriteWrapper_c&, bool, CSphString const&) in /usr/bin/searchd
5# CSphIndex_VLN::AddRemoveAttribute(bool, AttrAddRemoveCtx_t const&, CSphString&) in /usr/bin/searchd
6# RtIndex_c::AddRemoveAttribute(bool, AttrAddRemoveCtx_t const&, CSphString&) in /usr/bin/searchd
7# 0x000055AC652A216E in /usr/bin/searchd
8# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /usr/bin/searchd
9# ProcessSqlQueryBuddy(std::pair<char const*, int>, unsigned char&, ISphOutputBuffer&) in /usr/bin/searchd
10# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /usr/bin/searchd
11# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /usr/bin/searchd
12# 0x000055AC651E6584 in /usr/bin/searchd
13# Threads::CoRoutine_c::CreateContext(std::function<void ()>, VecTraits_T)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in /usr/bin/searchd
14# make_fcontext in /usr/bin/searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_1), proto mysql, state query, command alter_drop
--- Totally 3 threads, and 1 client-working threads ---
------- CRASH DUMP END -------

Additional context
Add any other context about the problem here.
In case you've faced a crash what indextool --check returns.
@tomatolog
Copy link
Contributor

could you provide your pupils index to reproduce issue here locally?

@FunkForever
Copy link
Contributor Author

CREATE TABLE pupils (
id bigint,
sexid integer,
hasgroups integer,
hasincontingent integer,
contingentlinktypeid integer,
educationtypeid integer,
servicecount integer,
birthdate bigint,
fullname string attribute indexed,
firstname string attribute,
lastname string attribute,
middlename string attribute,
sexname string attribute,
contingentguid string attribute,
vedomstvoids multi,
organizationids multi,
serviceids multi,
rbndoclassificationids multi,
rmspprivilegedata json,
privilegecategories json,
megarelationstatuses json,
statusesinvedom multi,
statusesinorg multi64
) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/var/lib/manticore/pupils/stopwords.txt' rt_mem_limit='536870912'

@sanikolaev
Copy link
Collaborator

I can't reproduce the crash using the below script neither in Centos 7 nor in MacOS:

➜  ~ cat /tmp/stopwords.txt
a
the


mysql> drop table if exists pupils; CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'; select * from pupils; ALTER TABLE pupils DROP COLUMN statusesinorg; desc pupils; drop table if exists pupils; CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'; insert into pupils(id) values(1); ALTER TABLE pupils DROP COLUMN statusesinorg; select id from pupils; drop table if exists pupils; CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'; insert into pupils(id) values(1); flush ramchunk pupils; ALTER TABLE pupils DROP COLUMN statusesinorg; select id from pupils; desc pupils;
--------------
drop table if exists pupils
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
select * from pupils
--------------

Empty set (0.00 sec)

--------------
ALTER TABLE pupils DROP COLUMN statusesinorg
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
desc pupils
--------------

+------------------------+--------+------------+
| Field                  | Type   | Properties |
+------------------------+--------+------------+
| id                     | bigint |            |
| fullname               | text   | indexed    |
| sexid                  | uint   |            |
| hasgroups              | uint   |            |
| hasincontingent        | uint   |            |
| contingentlinktypeid   | uint   |            |
| educationtypeid        | uint   |            |
| servicecount           | uint   |            |
| birthdate              | bigint |            |
| fullname               | string |            |
| firstname              | string |            |
| lastname               | string |            |
| middlename             | string |            |
| sexname                | string |            |
| contingentguid         | string |            |
| rmspprivilegedata      | json   |            |
| privilegecategories    | json   |            |
| megarelationstatuses   | json   |            |
| vedomstvoids           | mva    |            |
| organizationids        | mva    |            |
| serviceids             | mva    |            |
| rbndoclassificationids | mva    |            |
| statusesinvedom        | mva    |            |
+------------------------+--------+------------+
23 rows in set (0.00 sec)

--------------
drop table if exists pupils
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
insert into pupils(id) values(1)
--------------

Query OK, 1 row affected (0.00 sec)

--------------
ALTER TABLE pupils DROP COLUMN statusesinorg
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
select id from pupils
--------------

+------+
| id   |
+------+
|    1 |
+------+
1 row in set (0.00 sec)

--------------
drop table if exists pupils
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
CREATE TABLE pupils ( id bigint, sexid integer, hasgroups integer, hasincontingent integer, contingentlinktypeid integer, educationtypeid integer, servicecount integer, birthdate bigint, fullname string attribute indexed, firstname string attribute, lastname string attribute, middlename string attribute, sexname string attribute, contingentguid string attribute, vedomstvoids multi, organizationids multi, serviceids multi, rbndoclassificationids multi, rmspprivilegedata json, privilegecategories json, megarelationstatuses json, statusesinvedom multi, statusesinorg multi64 ) min_infix_len='2' index_exact_words='1' charset_table='0..9, A..Z->a..z, -, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451->U+0435, U+401->U+0415' blend_chars='U+0060,U+0027,U+2019,U+2018' morphology='stem_enru,soundex' stopwords='/tmp/stopwords.txt' rt_mem_limit='536870912'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into pupils(id) values(1)
--------------

Query OK, 1 row affected (0.00 sec)

--------------
flush ramchunk pupils
--------------

Query OK, 0 rows affected (0.03 sec)

--------------
ALTER TABLE pupils DROP COLUMN statusesinorg
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
select id from pupils
--------------

+------+
| id   |
+------+
|    1 |
+------+
1 row in set (0.00 sec)

--------------
desc pupils
--------------

+------------------------+--------+------------+
| Field                  | Type   | Properties |
+------------------------+--------+------------+
| id                     | bigint |            |
| fullname               | text   | indexed    |
| sexid                  | uint   |            |
| hasgroups              | uint   |            |
| hasincontingent        | uint   |            |
| contingentlinktypeid   | uint   |            |
| educationtypeid        | uint   |            |
| servicecount           | uint   |            |
| birthdate              | bigint |            |
| fullname               | string |            |
| firstname              | string |            |
| lastname               | string |            |
| middlename             | string |            |
| sexname                | string |            |
| contingentguid         | string |            |
| rmspprivilegedata      | json   |            |
| privilegecategories    | json   |            |
| megarelationstatuses   | json   |            |
| vedomstvoids           | mva    |            |
| organizationids        | mva    |            |
| serviceids             | mva    |            |
| rbndoclassificationids | mva    |            |
| statusesinvedom        | mva    |            |
+------------------------+--------+------------+
23 rows in set (0.00 sec)

Can you make sure the above reproduces the crash for you. Beware, it contains "drop table pupils".

@sanikolaev sanikolaev added bug waiting Waiting for the original poster (in most cases) or something else labels Apr 20, 2023
@FunkForever
Copy link
Contributor Author

Yes, it does. The problem that caused the drop was related to a broken index: when checking through indextool --check, errors were detected

@sanikolaev
Copy link
Collaborator

The problem that caused the drop was related to a broken index: when checking through indextool --check, errors were detected

We recently fixed a bug when an ALTER ADD COLUMN could cause to a corrupted index - 9c67742 . It could be your case.

I'm closing the issue, but feel free to reopen in case an ALTER still causes any crash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug waiting Waiting for the original poster (in most cases) or something else
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tomatolog @sanikolaev @FunkForever