Handle hyphens in user dir search porperly

[erikjohnston]

c.f. #16675

[reivilibre]

Somewhat feels like we should be tokenising ourselves when we add the vectors to Postgres as well (I think that's possible?), rather than trying to unify two systems that probably weren't meant to talk to each other.

But that is a fairly big migration to do, so this seems OK for now

[HarHarLinks]

according to https://spec.matrix.org/v1.10/appendices/#user-identifiers and even more so https://spec.matrix.org/v1.10/appendices/#historical-user-ids, this seems incorrect.

[erikjohnston]

Can you expand what the problem is? The output of the parsing has more to do with the way the DBs tokenise than the grammar of IDs?

[HarHarLinks]

please correct me if my process is wrong, but the tokenisation should match between synapse and db-internals, yes?

---

>>> _parse_words_with_icu("test.123")
['test.123']
>>> _parse_words_with_icu("test..123")
['test', '.123']

postgres=# SELECT * FROM ts_debug('simple', 'user.123');
 alias |    description    |  token   | dictionaries | dictionary |  lexemes   
-------+-------------------+----------+--------------+------------+------------
 file  | File or path name | user.123 | {simple}     | simple     | {user.123}
(1 row)

postgres=# SELECT * FROM ts_debug('simple', 'user..123');
   alias   |   description    | token | dictionaries | dictionary | lexemes 
-----------+------------------+-------+--------------+------------+---------
 asciiword | Word, all ASCII  | user  | {simple}     | simple     | {user}
 blank     | Space symbols    | ..    | {}           |            | 
 uint      | Unsigned integer | 123   | {simple}     | simple     | {123}
(3 rows)

---

>>> _parse_words_with_icu("test/123")
['test', '123']

postgres=# SELECT * FROM ts_debug('simple', 'user/123');
 alias |    description    |  token   | dictionaries | dictionary |  lexemes   
-------+-------------------+----------+--------------+------------+------------
 file  | File or path name | user/123 | {simple}     | simple     | {user/123}
(1 row)

---

>>> _parse_words_with_icu("test_123")
['test_123']

postgres=# SELECT * FROM ts_debug('simple', 'user_123');
   alias   |   description    | token | dictionaries | dictionary | lexemes 
-----------+------------------+-------+--------------+------------+---------
 asciiword | Word, all ASCII  | user  | {simple}     | simple     | {user}
 blank     | Space symbols    | _     | {}           |            | 
 uint      | Unsigned integer | 123   | {simple}     | simple     | {123}
(3 rows)

---

>>> _parse_words_with_icu("test+123")
['test', '123']

postgres=# SELECT * FROM ts_debug('simple', 'test+123');
   alias   |   description   | token | dictionaries | dictionary | lexemes 
-----------+-----------------+-------+--------------+------------+---------
 asciiword | Word, all ASCII | test  | {simple}     | simple     | {test}
 int       | Signed integer  | +123  | {simple}     | simple     | {+123}

---

legacy example:

>>> _parse_words_with_icu("test~123")
['test', '123']

postgres=# SELECT * FROM ts_debug('simple', 'test~123');
   alias   |    description    | token | dictionaries | dictionary | lexemes 
-----------+-------------------+-------+--------------+------------+---------
 asciiword | Word, all ASCII   | test  | {simple}     | simple     | {test}
 file      | File or path name | ~123  | {simple}     | simple     | {~123}
(2 rows)