user searched for $250 but found €250.

[ogolosovskiy]

Describe the bug
Weird search results, user searched for $250 but found €250.

To Reproduce

mysql> SELECT  txt FROM archiveogo_1 WHERE match('$250');
+-----------------+
| txt             |
+-----------------+
| $250 currency   |
| 250 currency    |
| €250 currency |
+-----------------+

Expected behavior

mysql> SELECT  txt FROM archiveogo_1 WHERE match('$250');
+-----------------+
| txt             |
+-----------------+
| $250 currency   |
+-----------------+

Describe the environment:
Manticore 6.0.2 89c7a51@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)
For Manticore 4 without columnar the same bug

• OS version:
  Debian GNU/Linux 11 (bullseye)

Messages from log files:
Logs without errors

My config -

IDX_TYPE
type = plain
charset_table = non_cjk, U+20BD, U+20AC
ngram_len=1
blend_chars = :, -, +, &, U+24, ., _
ngram_chars = cjk
columnar_attrs = username

[barryhunter]

You have $ in 'blend_chars'. So words get indexed both with and without the charactor. Hence matching both $250 and 250

As for the Euro symbol, check your encoding. Not entirely sure, but I think the Euro symbol pasted here is codepoint 128 (U+80) from Windows CP 1252, rather than U+20AC from unicode.

(but that Could be encoding issue on this forum. This page does say its UTF8)

[barryhunter]

Well it is kinda 'tricky' with keyword searching, as it designed for words, not numbers.

In concept if want to match with the currency symbol, would need it in charset_table. (and have to deal with different symbols, including different encodings)

... ie do also want to deal with if the text actually contains '250 USD' type thing? Or $2,000

Might need to do some normalization - eg with regex_filter, to get 'prices' into a consistent format. Also if want to deal with say $25.43 - which case might need . in charset table (which adds extra complexity as it also used for period!)

Frankly, if possible, might be best if can deal with the 'price' outside of 'keywords' eg using an actual numberic attribute, can also then store the currency in attribute too

SELECT * FROM index WHERE MATCH('word another') AND price = 250 AND currency = 'USD'

[tomatolog]

I can remove it from blend_chars but that won't help my users. The result will be the same|

Do you reindex your data after the change?
What is the output of the SELECT txt FROM archiveogo_1 WHERE match('$250'); SHOW META ?

Could you provide fully reproducible example (config, source data, query with output) ?

[sanikolaev]

Moving this issue to Discussions.

[ogolosovskiy]

Many thanks for the quick response.
Our users looking everything. We have Australian, Japanese, American, European customers :)
Its not accounting service, its messaging. I just trying to make user search results more friendly.

I have checked symbol €, its U+20AC.

I have added currency symbols to charset_table (and reindex all data from scratch).

IDX_TYPE = """type = plain
charset_table = non_cjk, U+20BD, U+20AC, U+24
ngram_len=1
blend_chars = :, -, +, &, ., _
ngram_chars = cjk
columnar_attrs = username
"""
But it dosent help. Removing blend_chars or changing ngram_len doesnt help too.

mysql> SELECT  txt, HIGHLIGHT() FROM archiveogo_1 WHERE match('$250') and username='578e85cc-a513-4e4a-8df9-0c8336c3fe86@conference.ogo.devintermedia.net'; show meta;
+-----------------+---------------------+
| txt             | highlight()         |
+-----------------+---------------------+
| 250 currency    | <b>250</b> currency |
| $250 currency   | $250 currency       |
| €250 currency | €250 currency     |
+-----------------+---------------------+
3 rows in set, 1 warning (0.07 sec)

+----------------+-----------------------------------------------------------------------------------------+
| Variable_name  | Value                                                                                   |
+----------------+-----------------------------------------------------------------------------------------+
| warning        | index archiveogo_1: query warning: modifiers must be applied to keywords, not operators |
| total          | 3                                                                                       |
| total_found    | 3                                                                                       |
| total_relation | eq                                                                                      |
| time           | 0.068                                                                                   |
| keyword[0]     | *250*                                                                                   |
| docs[0]        | 150                                                                                     |
| hits[0]        | 150                                                                                     |
| keyword[1]     | 250                                                                                     |
| docs[1]        | 1                                                                                       |
| hits[1]        | 1                                                                                       |
+----------------+-----------------------------------------------------------------------------------------+
11 rows in set (0.00 sec)

About reproducing -
Its 100% reproduce on Manticore 4 and 6. (I don't have the Manticore 5 installed to test).
Just add the strings "€250" and "$250" to your favorite test database.
Config - any by fact. I have not checked RT, but suppose its the same. My config above.
Query - SELECT txt, HIGHLIGHT() FROM index_name WHERE match('$250')