IndexTool claims that it doesn’t support field type: float_vector

[pavelnemirovsky]

Bug Description:

After performing an index check on a table with a Vector field type, I am encountering the following errors

FAILED, checking attribute 'vector'...
Unsupported header type: 10
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check FAILED, 60 failures reported, 66.4 sec elapsed
check FAILED, 60 failures reported, 66.4 sec elapsed

Manticore Search Version:

Manticore 6.3.6 593045790@24080214 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)

Operating System Version:

Ubuntu 20

Have you tried the latest development version?

No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[sanikolaev]

I can't reproduce it in the dev version:

➜  ~ mysql -P9306 -h0 -e "drop table if exists test; create table test ( title text, image_vector float_vector knn_type='hnsw' knn_dims='4' hnsw_similarity='l2' ); insert into test values ( 1, 'yellow bag', (0.653448,0.192478,0.017971,0.339821) ), ( 2, 'white bag', (-0.148894,0.748278,0.091892,-0.095406) ); flush ramchunk test; insert into test values ( 3, 'yellow bag', (0.653448,0.192478,0.017971,0.339821) );"
➜  ~ indextool --check test
Manticore 6.3.9 c96df395d@24112911 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

using config file '/opt/homebrew/etc/manticoresearch/manticore.conf'...
checking table 'test'...
WARNING: failed to load RAM chunks, checking only 1 disk chunks
checking schema...
checking disk chunk, extension 0, 0(1)...
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check passed, 0.0 sec elapsed
check passed, 0.0 sec elapsed

Might have been already fixed. If you can reproduce it in the dev version on your data - pls let me know.

[pavelnemirovsky]

@sanikolaev, the issue is still reproducible based on image: manticoresearch/manticore:dev

1. Create data as described in this ticket Query Results Inconsistent Across Nodes for an index #2779
2. Perform indextool for lisdocument1 index

root@manticore-local-lab:/var/lib/manticore# indextool --check lisdocument1 | grep -i FAILED -A 1
WARNING: failed to load RAM chunks, checking only 20 disk chunks
checking schema...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 1 failures reported, 0.2 sec elapsed
checking disk chunk, extension 2, 1(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 2 failures reported, 0.3 sec elapsed
checking disk chunk, extension 4, 2(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 3 failures reported, 0.4 sec elapsed
checking disk chunk, extension 5, 3(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 4 failures reported, 0.5 sec elapsed
checking disk chunk, extension 7, 4(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 5 failures reported, 0.6 sec elapsed
checking disk chunk, extension 8, 5(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 6 failures reported, 0.7 sec elapsed
checking disk chunk, extension 9, 6(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 7 failures reported, 0.8 sec elapsed
checking disk chunk, extension 10, 7(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 8 failures reported, 0.9 sec elapsed
checking disk chunk, extension 27, 8(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 9 failures reported, 1.1 sec elapsed
checking disk chunk, extension 13, 9(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 10 failures reported, 1.3 sec elapsed
checking disk chunk, extension 14, 10(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 11 failures reported, 1.4 sec elapsed
checking disk chunk, extension 25, 11(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 12 failures reported, 1.6 sec elapsed
checking disk chunk, extension 16, 12(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 13 failures reported, 1.7 sec elapsed
checking disk chunk, extension 26, 13(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 14 failures reported, 1.9 sec elapsed
checking disk chunk, extension 18, 14(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 15 failures reported, 2.0 sec elapsed
checking disk chunk, extension 19, 15(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 16 failures reported, 2.1 sec elapsed
checking disk chunk, extension 20, 16(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 17 failures reported, 2.2 sec elapsed
checking disk chunk, extension 21, 17(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 18 failures reported, 2.4 sec elapsed
checking disk chunk, extension 22, 18(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 19 failures reported, 2.5 sec elapsed
checking disk chunk, extension 23, 19(20)...
--
FAILED, 
Unsupported header type: 10
--
check FAILED, 20 failures reported, 2.6 sec elapsed
check FAILED, 20 failures reported, 2.6 sec elapsed

[sanikolaev]

Easier way to reproduce the issue:

snikolaev@dev2:~$ gzip -cd /home/snikolaev/lisdocument1.sql.gz | mysql -P9306 -h0
snikolaev@dev2:~$ sudo -u manticore indextool --check lisdocument1
Manticore 6.3.9 86e240c06@24121111 dev (columnar 2.3.1 edadc69@24112219) (secondary 2.3.1 edadc69@24112219)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
WARNING: table lisdocument1: table 'lisdocument1': morphology option changed from config has no effect, ignoring
checking table 'lisdocument1'...
WARNING: failed to load RAM chunks, checking only 2 disk chunks
checking schema...
checking disk chunk, extension 0, 0(2)...
WARNING: wordlist size mismatch (size=44, checkpoints=0)
checking schema...
checking dictionary...
checking data...
checking columnar storage...
	checking attribute 'id'...
	checked 28674/28674 docs
	ok
	checking attribute '$document_id_HASH'...
	checked 28674/28674 docs
	ok
	checking attribute 'document_id'...
	checked 28674/28674 docs
	ok
	checking attribute 'vector_dim'...
	checked 28674/28674 docs
	ok
FAILED, checking attribute 'vector'...
Unsupported header type: 10
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check FAILED, 1 failures reported, 0.0 sec elapsed
checking disk chunk, extension 1, 1(2)...
WARNING: wordlist size mismatch (size=44, checkpoints=0)
checking schema...
checking dictionary...
checking data...
checking columnar storage...
	checking attribute 'id'...
	checked 49796/49796 docs
	ok
	checking attribute '$document_id_HASH'...
	checked 49796/49796 docs
	ok
	checking attribute 'document_id'...
	checked 49796/49796 docs
	ok
	checking attribute 'vector_dim'...
	checked 49796/49796 docs
	ok
FAILED, checking attribute 'vector'...
Unsupported header type: 10
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check FAILED, 2 failures reported, 0.2 sec elapsed
check FAILED, 2 failures reported, 0.2 sec elapsed