min_infix_len

[Buffele]

Hi!

I'm a long time sphinxsearch user, currently using sphinxsearch 2.x.
I would love to upgrade to manticoresearch but currently

WARNING: min_infix_len must be greater than 1, changed to 2

this is holding us back.

We are using

min_infix_len = 1

in production since around 6 years on sphinxsearch and never had performance issues.

Could we maybe make manticoresearch compatible with min_infix_len = 1 ? I would be fine with a big red warning in the logs too.
I'm sure my use case is extremely specific but we need min_infix_len = 1 and I would want to avoid to fork manticoresearch.

[sanikolaev]

Hi. Thank you for the feedback. We'll discuss it with them team.

[Buffele]

Thank you very much.

I know our use case is extremely specific, since we have a dataset of around 300k entries with 1 word entries only. And we need a autocomplete / search suggestions as soon as the user starts typing, so also with one character. Works flawlessly with sphinxsearch 2.2.

Sadly now, no other search engine really supports it properly. That's why we're stuck with this extremely old sphinxsearch version.

[tomatolog]

could you check that

min_infix_len = 2
min_prefix_len = 1

would work for you?

[Buffele]

Where exactly? In the sphinxsearch version I'm using or in the latest manticoresearch?

[sanikolaev]

In Manticore. The point is that you perhaps want to be able to do: a* and *ab*, but not *a*. It works fine in Manticore.

[Buffele]

That's the thing. Business requirements require me to do a *a*. a* does not suffice here.

[sanikolaev]

@Buffele I've discussed this with the team. You may be mistaken that min_infix_len = 1 is working in Sphinx, it may just fallback to the value of 2 exactly like in Manticore. The developers (ex-Sphinx developers) don't remember they ever implemented a single-char support in Sphinx, the algorithm should be based on bigrams. I also can't see the support of min_infix_len = 1 in Sphinx docs. Can you check it carefully?

[Buffele]

Let me check my implementation and then get back to you. I'm pretty sure it works; I will create a demo repository with all exact specifications.

[Buffele]

Okay, just checked.

I'm using sphinx version Sphinx 2.1.9-id64-release (r4761). I don't see any mentions of dict in my configuration file. Could it be that the default in this version was dict=crc, hence the min_infix_len = 1 working?

Ah, i just checked.

Configuration looks like that for this version:

# dictionary type, 'crc' or 'keywords'
# crc is faster to index when no substring/wildcards searches are needed
# crc with substrings might be faster to search but is much slower to index
# (because all substrings are pre-extracted as individual keywords)
# keywords is much faster to index with substrings, and index is much (3-10x) smaller
# keywords supports wildcards, crc does not, and never will
# optional, default is 'crc'
dict			= keywords

So I was always using dict=crc, hence it worked flawlessly.

[barryhunter]

From what I remember min_infix_len=1 works with dict=crc. It does not work with dict=keywords.

In fact just tested on https://play.manticoresearch.com/maindelta/ (as I remember that config has a nice small index to test with)

Edited the config to have

index main
{
        path = /var/lib/manticore/main
        source = main

        min_infix_len = 1
        dict=crc
}

(also modified sql_range_step to have a much larger value, otherwise the building the index is REALLY slow, large timestamp differences now!)

And it seems to work

mysql> select * from main;
+------+----------------+------------+
| id   | title          | updated    |
+------+----------------+------------+
|    1 | Document One   | 1653475426 |
|    2 | Document Two   | 1653475426 |
|    3 | Document Three | 1653475426 |
|    4 | Document Four  | 1653475426 |
|    5 | Document Five  | 1653475426 |
|    6 | Document Six   | 1653475426 |
|    7 | Document Seven | 1653475426 |
|    8 | Document Eight | 1653475426 |
|    9 | Document Nine  | 1653475426 |
+------+----------------+------------+
9 rows in set (0.01 sec)

mysql> select * from main where match('*i*');
+------+----------------+------------+
| id   | title          | updated    |
+------+----------------+------------+
|    5 | Document Five  | 1653475426 |
|    6 | Document Six   | 1653475426 |
|    8 | Document Eight | 1653475426 |
|    9 | Document Nine  | 1653475426 |
+------+----------------+------------+
4 rows in set (0.00 sec)

mysql> show meta;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| total         | 4     |
| total_found   | 4     |
| time          | 0.006 |
| keyword[0]    | *i*   |
| docs[0]       | 4     |
| hits[0]       | 4     |
+---------------+-------+
6 rows in set (0.00 sec)

[barryhunter]

To be clear, dict=keywords is great for normal 'whole word' matching. But infix/prefix - practically for short keywords, dict=crc is the winner IMHO.

[Buffele]

Okay, thanks for the explanation.

dict=crc sounds perfect for my specific use case then because all data I'm indexing is one word, barely any spaces. Total character length of 30 at most. And the table has like 500k entries. I expect the table to grow to 1 million entries but even this should suffice with performance.

I guess my old sphinx version (Sphinx 2.1.9-id64-release (r4761)) was using dict=crc as default then.

[barryhunter]

Ah yes, the default did change at some point, so you may of had dict=crc by default without realizing.

Just for completeness, dict=keyword has other advantages, in particular CALL SUGGEST() support, and being able to get the list of words matching a wildcard term (from CALL KEYWORDS()). Or even dump the keyword list with indextool from an existing index. Can't do any of that with dict=crc. To get a list of words has to be done at indexing time (pre-hashing).

[Buffele]

Since I'm apparently using dict=crc for years already and never missed any features except for having the software updated, I don't think I need that. My current search engine requirements are perfectly covered. Also the lack of RT tables is something I don't really miss. In fact, I like the re-indexing much more, so you can make sure it indexes everything and is never out of sync (except in the time waiting for the next sync, which is fine). I'm still baffled that only this search engine is providing a ready to use indexer, all other search engines require you to feed data through their HTTP API endpoints. (Which is much more work)

[barryhunter]

Yes, 'indexer' is one of the things that drew me to sphinx originally. Great for quickly building indexes.

It can be tricky to 'scale', ie rebuilding the whole index each time is 'inefficient', as all data needs to be reprocessed, even if unchanged.

The relative simplicity of indexer, makes it robust. As you say can be quite confident it getting everything.

[Buffele]

My last question here would be then:

There is no plans anytime soon or in the future to deprecate or even remove dict=crc, right?

Would be great if manticoresearch can cover this special let's call it "single words autocomplete suggestions" use case perfectly.

[sanikolaev]

I confirm the core team doesn't have plans to deprecate dict=crc or plain tables.