Speller suggestion issue

[snomos]

@Trondtr has reported:

echo а̄им | hfst-ospell -S tools/spellcheckers/mns.zhfst
"а̄им" is NOT in the lexicon:
Corrections for "а̄им":
вим    20.056900
аим    20.787788
йим    21.743298
мим    22.254124
сым    22.524096
оим    22.659590

Now compare this with the following:

grep а tools/spellcheckers/strings.default.txt
а:а̄    1
#а̄:я   4
а̄:а    1
ся:ща   2
Ся:Ща   2

Why is ами suggested as second?

The core of the issue is that а̄ has a base char + combining macron: how well / bad does the error model handle combining diacritics when it comes to suggestions?

[snomos]

The interesting thing happens when you use divvunspell instead of hfst-ospell:

echo а̄им | divvunspell suggest -a tools/spellcheckers/mns.zhfst
Reading from stdin...
Input: а̄им		[INCORRECT]
аим		17.787788
а̄тим		24.199446
а̄гим		25.03525
аким		26.155512
ам		27.999014
агим		28.65959
атым		32.826374
аюм		34.254124
аи		43.7433
аис		43.966442

For comparison, hfst-ospell-office gives the same output as hfst-ospell:

echo '5	а̄им' | hfst-ospell-office -d tools/spellcheckers/mns.zhfst
@@ Loading tools/spellcheckers/mns.zhfst with args max-weight=-1.00, beam=-1.00, time-cutoff=6.00
@@ hfst-ospell-office is alive
&	вим (20.06;0)	аим (20.79;0)	йим (21.74;0)	мим (22.25;0)	сым (22.52;0)

[snomos]

@flammie do you have comments or insights re combining diacritics and hfst-ospell?

In any case: I am not sure how much time we should spend on this, given that it works correct using divvunspell, and divvunspell is used everywhere except in the grammar checker — and from the summer also in the grammar checker. That is, hfst-ospell is not first priority.

[flammie]

yeah it seems quite fragile here at compilation of separate error model part already:

$ echo 'а̄' | hfst-lookup .generated/strings.all.default.hfst 
hfst-lookup: warning: It is not possible to perform fast lookups with OpenFST, std arc, tropical semiring format automata.
Using HFST basic transducer format and performing slow lookups
> а̄	а̄̄	1,000000
а̄	а	2,000000
$ hfst-fst2txt .generated/strings.all.default.hfst | fgrep а
0	0	а	а	0.000000
1	21	а	а	0.000000
2	43	я	а	0.000000
41	44	я	а	0.000000
58	58	а	а	0.000000
$  hfst-fst2txt .generated/strings.all.default.hfst | fgrep а̄
$  hfst-fst2txt .generated/strings.all.default.hfst | fgrep $'\u0304'
0	0			0.000000
7	8			0.000000
13	14	@0@		0.000000
13	27		@0@	0.000000
15	16	@0@		0.000000
15	28		@0@	0.000000
17	18	@0@		0.000000
17	29		@0@	0.000000
19	20	@0@		0.000000
19	30		@0@	0.000000
21	22	@0@		0.000000
21	31		@0@	0.000000
23	24	@0@		0.000000
23	32		@0@	0.000000
25	26	@0@		0.000000
25	33		@0@	0.000000
58	58			0.000000
$  hfst-fst2txt .generated/strings.all.default.hfst
0	0	@_IDENTITY_SYMBOL_@	@_IDENTITY_SYMBOL_@	0.000000
0	0			0.000000
0	0	С	С	0.000000
0	0	Щ	Щ	0.000000
0	0	а	а	0.000000
0	0	г	г	0.000000
0	0	е	е	0.000000
0	0	и	и	0.000000
0	0	й	й	0.000000
0	0	к	к	0.000000
0	0	н	н	0.000000
0	0	о	о	0.000000
0	0	р	р	0.000000
0	0	с	с	0.000000
0	0	т	т	0.000000
0	0	у	у	0.000000
0	0	щ	щ	0.000000
0	0	ы	ы	0.000000
0	0	ь	ь	0.000000
0	0	э	э	0.000000
0	0	ю	ю	0.000000
0	0	я	я	0.000000
0	0	ӈ	ӈ	0.000000
0	0	ӣ	ӣ	0.000000
0	0	ӯ	ӯ	0.000000
0	1	@0@	@0@	0.000000
1	2	с	щ	0.000000
1	4	т	к	0.000000
1	9	т	т	0.000000
1	13	я	я	0.000000
1	15	э	э	0.000000
1	17	ы	ы	0.000000
1	19	ю	ю	0.000000
1	21	а	а	0.000000
1	23	е	е	0.000000
1	25	о	о	0.000000
1	34	н	ӈ	0.000000
1	36	ӈ	н	0.000000
1	38	г	ӈ	0.000000
1	41	С	Щ	0.000000
1	45	н	н	0.000000
1	48	ӯ	ӯ	0.000000
1	51	у	у	0.000000
2	3	ь	@0@	0.000000
2	40	ю	у	0.000000
2	43	я	а	0.000000
3	58	@0@	@0@	1.000000
4	5	и	и	0.000000
4	6	ӣ	ӣ	0.000000
4	7	е	е	0.000000
5	58	@0@	@0@	2.000000
6	58	@0@	@0@	2.000000
7	8			0.000000
7	58	@0@	@0@	2.000000
8	58	@0@	@0@	2.000000
9	10	т	к	0.000000
10	11	е	е	0.000000
10	12	и	и	0.000000
11	58	@0@	@0@	2.000000
12	58	@0@	@0@	2.000000
13	14	@0@		0.000000
13	27		@0@	0.000000
14	58	@0@	@0@	1.000000
15	16	@0@		0.000000
15	28		@0@	0.000000
16	58	@0@	@0@	1.000000
17	18	@0@		0.000000
17	29		@0@	0.000000
18	58	@0@	@0@	1.000000
19	20	@0@		0.000000
19	30		@0@	0.000000
20	58	@0@	@0@	1.000000
21	22	@0@		0.000000
21	31		@0@	0.000000
22	58	@0@	@0@	1.000000
23	24	@0@		0.000000
23	32		@0@	0.000000
24	58	@0@	@0@	1.000000
25	26	@0@		0.000000
25	33		@0@	0.000000
26	58	@0@	@0@	1.000000
27	58	@0@	@0@	2.000000
28	58	@0@	@0@	2.000000
29	58	@0@	@0@	2.000000
30	58	@0@	@0@	2.000000
31	58	@0@	@0@	2.000000
32	58	@0@	@0@	2.000000
33	58	@0@	@0@	2.000000
34	35	г	@0@	0.000000
35	58	@0@	@0@	2.000000
36	37	@0@	г	0.000000
37	58	@0@	@0@	2.000000
38	39	н	н	0.000000
39	58	@0@	@0@	3.000000
40	58	@0@	@0@	2.000000
41	42	ю	у	0.000000
41	44	я	а	0.000000
42	58	@0@	@0@	2.000000
43	58	@0@	@0@	2.000000
44	58	@0@	@0@	2.000000
45	46	т	р	0.000000
46	47	р	@0@	0.000000
47	58	@0@	@0@	4.000000
48	49	й	и	0.000000
48	54	й	ы	0.000000
49	50	и	@0@	0.000000
50	58	@0@	@0@	4.000000
51	52	й	и	0.000000
51	56	й	ы	0.000000
52	53	и	@0@	0.000000
53	58	@0@	@0@	4.000000
54	55	ы	@0@	0.000000
55	58	@0@	@0@	4.000000
56	57	ы	@0@	0.000000
57	58	@0@	@0@	4.000000
58	58	@_IDENTITY_SYMBOL_@	@_IDENTITY_SYMBOL_@	0.000000
58	58			0.000000
58	58	С	С	0.000000
58	58	Щ	Щ	0.000000
58	58	а	а	0.000000
58	58	г	г	0.000000
58	58	е	е	0.000000
58	58	и	и	0.000000
58	58	й	й	0.000000
58	58	к	к	0.000000
58	58	н	н	0.000000
58	58	о	о	0.000000
58	58	р	р	0.000000
58	58	с	с	0.000000
58	58	т	т	0.000000
58	58	у	у	0.000000
58	58	щ	щ	0.000000
58	58	ы	ы	0.000000
58	58	ь	ь	0.000000
58	58	э	э	0.000000
58	58	ю	ю	0.000000
58	58	я	я	0.000000
58	58	ӈ	ӈ	0.000000
58	58	ӣ	ӣ	0.000000
58	58	ӯ	ӯ	0.000000
58	0.000000

[snomos]

Do I read the above correct, @flammie, when I find no occurrences of а̄ in the ATT version of strings.all.default.hfst? So the base char + combining diacritic is lost during compilation?

If so, how can we force such a sequence to be treated as one symbol, in all contexts? The lexical FST does treat them as one symbol (as opposed to the tokeniser FST, which does the opposite on the input side).

I assume the question relates to all .txt input files for the error model.

[flammie]

Mm, strings compilation uses hfst-strings2fst just without any alphabets / multichars so it must consider combining characters their own arcs in the graph. I guess it leads into situation where suggestions from а̄им to вим weighs а:в and to аим weighs combining macron:0 rather. Maybe applying nfc/nfd filters in error models could work if this is actually the main issue

[snomos]

mm, that might be a good idea. I will have a look.

[snomos]

giellalt/giella-core@c73d62a fixes a bug that hindered spellrestrict.* files from being built. But that is not enough, the generated spellrestrict.regex file does not include the relevant letters.

[snomos]

Ah - bummer on my part. The spellrestrict.* files will not solve this, because they assume the existence of an NFC form. In this case the problem is that there IS NO NFC form of the relevant letters, but we still want the FST to treat them as one symbol, ie no arch between base letter and combining diacritics.

So we need a new type of filter that finds all combining diacritics and the corresponding base letter(s), and then generates a filter of the following type:

{а̄} -> "а̄";

and then applies this filter to all error model files being read by hfst-strings2fst on both sides. That should hopefully fix the issue, and make it predictable to work with combining diacritics in the speller.

@flammie feel free to continue this work 🙂