Message /search API : Support CJK full text search

[GrassBlock1]

Clients like cinny implements message (room events) search via the /_matrix/client/v3/search api, and it works out of the box for English and other language using the space to divide the words. But when message was written in Chinese (Korean and Japanese), conduwuit only returns the message that starts with the word.

For example, when searching for "可能", it only returns message like this:

and messages like "还有可能是定期检修" can't be found by this method.

I also tried to look up the conduwuit documentation but seems that it's not mentioned.

I would like to know if there are solutions for this kind of messages as I use Chinese as my main language and it will truly make experience better for users who use them too.

[Kimiblock]

Use clients that implement searching themselves, like nheko. AFAIK Synapse also has this problem.

[GrassBlock1]

Use clients that implement searching themselves, like nheko. AFAIK Synapse also has this problem.

Oh that's a way.
Since I have account using nheko and it doesn't support multiple accounts, (fluffychat also supports this but it was laggy on my pc) I had to use cinny.

keep open in hope there will be better support for that.

[Kimiblock]

nheko --help | grep profile

[GrassBlock1]

nheko --help | grep profile

Oh it works well for me, thanks!

[girlbossceo]

The text search implementation we inherited is pretty bad, so it's likely that a whole new system would need to be developed. Synapse has the same problems and has a poor text search implementation too.

[GrassBlock1]

The text search implementation we inherited is pretty bad, so it's likely that a whole new system would need to be developed. Synapse has the same problems and has a poor text search implementation too.

I see.
Is there any plans on it?

[girlbossceo]

Not in the near future, which is why I marked it as no priority, especially with the various other higher priority stuff we have going on.

It's a very huge undertaking of developing an in-house full text search implementation, and arguably maybe it shouldn't even be our problem and let something like ElasticSearch or whatever deal with it.

[GrassBlock1]

Not in the near future, which is why I marked it as no priority, especially with the various other higher priority stuff we have going on.

OK, thanks for your consideration.

It's a very huge undertaking of developing an in-house full text search implementation, and arguably maybe it shouldn't even be our problem and let something like ElasticSearch or whatever deal with it.

Yes, I think it's a way to let external search engine to manage these stuff since I have got Meilisearch working and it works well for CJK search.