fix: fix custom wordbreaker output format for sil.km.gcc

[jahorton]

Fixes keymanapp/keyman#12200.

Recent changes in 18.0-alpha require an update to the custom wordbreaker I originally wrote for this model. We now seek to model whitespaces in the context to ensure we don't accidentally auto-correct away punctuation marks, but those same changes require precise adherence to the wordbreaker output format. Unfortunately, my original version of it played a bit... fast and loose, returning improper values for blank or duplicated tokens.

[jahorton]

The key part of the fix: we should track the furthest index reached while iterating over each token and skip past it when determining the original position of the next token to be processed. This fixes handling for any duplicate-string cases.

[jahorton]

Prevents empty tokens from appearing in the middle of context, especially should excessive whitespaces exist between words. Something like the quick brown fox should appear to skip all spaces between tokens at once, not in multiple steps.

[darcywong00]

Does the custom wordbreaker for sil.km.cnd.model.ts need a similar fix?

lexical-models/release/sil/sil.km.cnd/source/sil.km.cnd.model.ts

Lines 13 to 25 in a90fec5

	wordBreaker: function(str: string) {
	return str.split(/\s|\u200b/).map(function(token) {
	return {
	left: str.indexOf(token),
	start: str.indexOf(token),
	right: str.indexOf(token) + token.length,
	end: str.indexOf(token) + token.length,
	text: token
	}
	});
	},
	punctuation: {
	insertAfterWord: "\u200B"

[DavidLRowe]

@darcywong00 Would you be able to review this PR? I don't think I'm qualified.

[jahorton]

For what it's worth, I've included a copy of the new wordbreaker code in keymanapp/keyman#12229, which is then used in automated tests. See https://github.com/keymanapp/keyman/blob/ec32e21bf4bd6f7d225126e34deaec7ff728e302/common/models/templates/test/test-tokenization.js#L529-L575.

The automated test spec before it tests a mitigation for the old version and includes comments talking about the issue we ran into that motivated this. But yeah, there are some predictive-text internal details that might make a review a bit tricky. Just thought I'd provide the most relevant references in case whoever does review might need 'em.

[darcywong00]

Recent changes in 18.0-alpha require an update to the custom wordbreaker I originally wrote for this model.

So should these changes get in before we rebuild all the lexical-models in #246?
e.g. Should we bump the gcc-custom-breaker version (in the .kps) here or #246?

[jahorton]

Recent changes in 18.0-alpha require an update to the custom wordbreaker I originally wrote for this model.

So should these changes get in before we rebuild all the lexical-models in #246? e.g. Should we bump the gcc-custom-breaker version (in the .kps) here or #246?

Either order would likely be fine, but yeah, we'd probably need to bump the version again if done after #246, assuming it includes a version bump. So... it wouldn't hurt to push this in before the "rebuild all".