Short time format is missing space under en_US on Fedora 38+

[tmds]

The short time format on Fedora 38 has replaced its breaking space by a 'NARROW NO-BREAK SPACE' (U+202F).

That space gets removed by:

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureData.Icu.cs

Line 225 in 34c0472

private static string ConvertIcuTimeFormatString(ReadOnlySpan<char> icuFormatString)

Any character not explicitly recognized by this function (like U+202F) gets removed.

This function includes a specific case for a regular 'NO-BREAK SPACE' (U+00A0):

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureData.Icu.cs

Lines 261 to 264 in 34c0472

	case '\u00A0':
	// Convert nonbreaking spaces into regular spaces
	result[resultPos++] = ' ';
	break;

There are two ways to fix this:

• Instead of removing unknown characters, we pass characters (including these non-breaking spaces) as is.
• Or, we add U+202F so it also gets converted to a regular space.

I don't know why the current implementation opted for the second option for U+00A0.
I think it may be to have the same time format as on Windows under en_US.

I have a slight preference for the first option, because the second is overwriting part of the format information from icu. And, the second option would have prevented this issue from occurring.

What is the preferred option?

cc @omajid

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The short time format on Fedora 38 has replaced its breaking space by a 'NARROW NO-BREAK SPACE' (U+202F).

That space gets removed by:

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureData.Icu.cs

Line 225 in 34c0472

private static string ConvertIcuTimeFormatString(ReadOnlySpan<char> icuFormatString)

Any character not explicitly recognized by this function (like U+202F) gets removed.

This function includes a specific case for a regular 'NO-BREAK SPACE' (U+00A0):

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/CultureData.Icu.cs

Lines 261 to 264 in 34c0472

	case '\u00A0':
	// Convert nonbreaking spaces into regular spaces
	result[resultPos++] = ' ';
	break;

There are two ways to fix this:

• Instead of removing unknown characters, we pass characters (including these non-breaking spaces) as is.
• Or, we add U+202F so it also gets converted to a regular space.

I don't know why the current implementation opted for the second option for U+00A0.
I have a slight preference for the first option, because the second is overwriting part of the format information from icu.

What is the preferred option?

cc @omajid

Author:	tmds
Assignees:	-
Labels:	area-System.Globalization
Milestone:	-

[tmds]

cc @stephentoub

[stephentoub]

I don't know why the current implementation opted for the second option for U+00A0.
I think it may be to have the same time format as on Windows under en_US.

@ellismg, do you remember?

Instead of removing unknown characters, we pass characters (including these non-breaking spaces) as is.

What breaks if we do that?

[tmds]

What breaks if we do that?

It affects formatting: non-breaking spaces don't break to the next line.

It breaks tests that expect the space to be used on Linux in the en-US time format, like on Windows.

I was wondering about something.

using System.Text;

foreach (string s in new string[] { "'\u202F'", "'\u00A0'" })
{
    Console.WriteLine(s);
    Console.WriteLine(AsciiRoundTrip(s));
}

static string AsciiRoundTrip(string s) =>
    Encoding.ASCII.GetString(Encoding.ASCII.GetBytes(s));

This prints:

' '
'?'
' '
'?'

It seems neither of these non-breaking spaces get converted to regular spaces when converting to ASCII.
If that's the intended behavior for the ASCII conversion(?), it seems desired to not include non-breaking spaces for ASCII compatibility.

[tmds]

Based on the above, I assume that, for ASCII conversion purposes, we want regular spaces in the time format.

[Clockwork-Muse]

If that's the intended behavior for the ASCII conversion(?), it seems desired to not include non-breaking spaces for ASCII compatibility.

Yeah, but that's ASCII. Format strings can have any number of non-ASCII characters (eg, the year/month/day symbols in Japanese), so doing it at the ICU ingest step seems wrong.

[tmds]

so doing it at the ICU ingest step seems wrong.

I agree.
I would also prefer we use the format strings as they come from icu.
That is the correct choice.

The non-breaking space case was already there.
The rationale isn't mentioned in the code. It's my guess it's for using a space that exists in ASCII.
Adding another type of non-breaking space is the safe choice as it minimizes breaking anyone.

Format strings can have any number of non-ASCII characters

Any such characters are currently filtered out in ConvertIcuTimeFormatString.

[stephentoub]

I would also prefer we use the format strings as they come from icu. That is the correct choice.

You're suggesting we have two completely different parsing / formatting code paths based on whether the format string came from a user or from ICU? What about places where these format strings are returned to the user and need to be the .NET format? I don't see how this is a viable or correct choice. Have I totally misunderstood?

[tmds]

This function takes the format string that comes from icu, and translate it in to the equivalent .NET format string.