Allow ANCM in-process to interpret request headers as Latin1 encoded

[jkotalik]

Similar issue with Kestrel: #17399

Related PR for the fix in Kestrel: #18255

This is a very similar ask that we had for Latin1 support in Kestrel.

In a 3.1 patch we should provide:

• A config setting to change the default encoding to Latin1
• When enabled, ANCM will widen each incoming byte to a UTF-16 character and build a string from that set of characters, instead of interpreting bytes as UTF-8.
• We will continue to reject control characters from the ASCII set (0x00-0x1F and 0x7F), but not reject control characters from the widened Latin1 set (because they would collide with other interpretations of this data, like UTF-8).

This would allow a consumer to reinterpret the data in this string as a UTF-8 sequence if desired.

In 5.0 we'll look at broader work to improve this experience. The 3.1 goals are scoped down to specific requests we've received.

[jkotalik]

cc @Pilchie @halter73 @Tratcher

[Pilchie]

Thanks - do you have an estimate of how much work this is? Is this something we could consider for July servicing?

[jkotalik]

My guess would be around 2 days of work. We should be able to piggy back off the work done in Kestrel. Reaching out to partners for verification isn't included in this cost.

[jkotalik]

Costing as medium though as things always take longer than expected 😄

[Tratcher]

• A config setting to change the default encoding to Latin1

This probably needs to be an AppContext switch.

In 5.0 we'll look at broader work to improve this experience.

See #17400

[Pilchie]

Okay - can we start working on this soon?

[jkotalik]

I could get started on this Friday (will be occupied with tye things until then). Is that timeline okay for you?

[Pilchie]

Yes, sounds good. Thanks @jkotalik.