Unicode German sharp S violates an assumption of some string methods

[dmk42]

Chapel strings are currently immutable except for appending, and upper/lower case conversion. In case conversion, the methods assume that the upper and lower case versions of a letter encode to the same number of UTF-8 bytes. While documenting some properties of strings, I came across a letter that violates that assumption.

Unicode's German lower-case sharp S (ß) was encoded in 1985 (U+00DF) in an earlier character set and eventually absorbed into Unicode.

Unicode's German upper-case sharp S (ẞ) was encoded in 2008 (U+1E9E).

Consequently, the lower case version translates to two bytes in UTF-8, and the upper case version translates to three bytes.

Darwin's towlower() and towupper() routines compensate by refusing to change the case of a sharp S. Cygwin's and Linux's towupper() will not change the case of a lower case sharp S (preventing buffer overflows) but their towlower() will change the case of an upper case sharp S, generating a character that takes one less byte than before.

Eventually it would be a good idea to have the String module change the byte length of strings as necessary to accommodate any differences in the length of the upper vs. lower case characters. Alternatively, the String module could enforce the Darwin approach everywhere, and refuse to change the case of any character where that would change the number of bytes. Another alternative is for the String module to enforce the Linux approach everywhere, and refuse to change the case of any character where that would take more bytes than before.

The affected string methods are the following.

toLower()
toUpper()
toTitle()
capitalize()

#13649 adds a future to track the bug.

[dmk42]

@e-kayrakli - FYI.