Add StringContent ctor providing tighter control over charset

[guardrex]

API Approval - iteration 2

This is being submitted again for approval because we decided additional APIs were needed.
Copied from proposal in #17036 (comment)

public class StringContent
{
   public StringContent(string content);
+  public StringContent(string content, MediaTypeHeaderValue mediaType); // Already approved in iteration 1
   public StringContent(string content, Encoding encoding);
   public StringContent(string content, Encoding encoding);
   public StringContent(string content, Encoding encoding, string mediaType);
+  public StringContent(string content, Encoding encoding, MediaTypeHeaderValue mediaType); // NEW API proposal
}

public class MediaTypeHeaderValue
{
    public MediaTypeHeaderValue(string mediaType)
+   public MediaTypeHeaderValue(string mediaType, string charSet);  // NEW API proposal
}

Previous API approval is here: #17036 (comment)

Original post

Consider the following code ...

string contentType = "application/atom+xml";
string requestPayload = "some data";
var stringContent = new StringContent(requestPayload, Encoding.UTF8, contentType);

According to Wireshark, I'm seeing a POST using HttpClient with this ...

Content-Type: application/atom+xml; charset=utf-8

Shouldn't that Content-Type value just be ...

Content-Type: application/atom+xml

?? Confused, because something is choking my Azure Table Storage requests with ...

string stringToSign = 
    $"{requestMethod}\n\n{contentType}\n{dateInRfc1123Format}\n{canonicalizedResource}";

... and since SharedKeyLite, which only requires the date and resource, works with my dateInRfc1123Format and canonicalizedResource, I've sort of narrowed it down to the contentType of the request (shown above) not matching what I'm putting into the signature sting, namely just application/atom+xml.

[benaadams]

Looks like it will always put something out https://github.com/dotnet/corefx/blob/d0dc5fc099946adc1035b34a8b1f6042eddb0c75/src/System.Net.Http/src/System/Net/Http/StringContent.cs#L29

[davidsh]

The StringContent constructor follows the same behavior as .NET Framework (Desktop).

You can always replace the Content-Type (or any header) with something else if the default behavior isn't what you want.

[JonHanna]

It's certainly invalid. RFC 2046 (RFC 6657 is also relevant) allows for the charset parameter as part of content-types only if it's text/plain or a text/* type that specifies it too allows.

[guardrex]

@benaadams @JonHanna Thanks for finding the references to both. Even if it were valid (e.g., text/plain or text/*), I think if you supply the value explicitly, the value should be honored when the header is created IMHO.

I don't know if this is what is choking my ATS REST service calls, but it's possible, since SharedKeyLite works with my date and resource. This is certainly a suspect for my failing SharedKey request failures.

@davidsh How? I get an exception if I try to mod the header by adding ...

client.DefaultRequestHeaders.Add("Content-Type", contentType);

... throws ...

System.InvalidOperationException: Misused header name. Make sure request headers are used 
with HttpRequestMessage, response headers with HttpResponseMessage, and content headers 
with HttpContent objects.
 at System.Net.Http.Headers.HttpHeaders.CheckHeaderName(String name)
 at teststandalone.HomeController.DoIt()

Here's the hacked code I'm playing with (i.e., not for production ... I'm just hacking around to get the requests working). How would I set the Content-Type explicitly to override the StringContent one given this code ..

using (var client = new HttpClient())
{
    DateTime requestDT = DateTime.UtcNow;
    UTF8Encoding utf8Encoding = new UTF8Encoding();
    string storageAccountName = "<STORAGE_ACCOUNT_NAME>";
    string tableName = "<TABLE_NAME>";
    //string requestMethod = "POST";
    string dateInRfc1123Format = requestDT.ToString("R", CultureInfo.InvariantCulture);
    string storageServiceVersion = "2015-04-05";
    string storageAccountKey = "<STORAGE_ACCOUNT_KEY>";
    string canonicalizedResource = $"/{storageAccountName}/{tableName}";
    string contentType = "application/atom+xml";

    // Content
    string requestPayload = GetRequestContentInsertXml(requestDT, "PKEY", "RKEY");
    var stringContent = new StringContent(requestPayload, Encoding.UTF8, contentType);

    // Will not authorize against the service:
    // string stringToSign = $"{requestMethod}\n\n{contentType}\n{dateInRfc1123Format}\n{canonicalizedResource}";
    // SharedKeyLite works:
    string stringToSign = $"{dateInRfc1123Format}\n{canonicalizedResource}";
    string authorizationHeader = CreateAuthorizationParameter(stringToSign, storageAccountName, storageAccountKey);
    AuthenticationHeaderValue myAuthHeaderValue = new AuthenticationHeaderValue("SharedKeyLite", authorizationHeader);

    // Headers
    client.DefaultRequestHeaders.Authorization = myAuthHeaderValue;
    client.DefaultRequestHeaders.Add("x-ms-date", dateInRfc1123Format);
    client.DefaultRequestHeaders.Add("x-ms-version", storageServiceVersion);
    client.DefaultRequestHeaders.Add("DataServiceVersion", "3.0;NetFx");
    client.DefaultRequestHeaders.Add("MaxDataServiceVersion", "3.0;NetFx");
    // Doesn't work: client.DefaultRequestHeaders.Add("Content-Type", contentType);

    var responseMessage = client.PostAsync($"https://{storageAccountName}.table.core.windows.net/{tableName}", stringContent).Result;

    responseResult.Append("StatusCode: " + responseMessage.StatusCode.ToString() + " ");

}

[davidsh]

When modifying "known" headers, you need to use the properties and not the 'DefaultRequestHeaders' method. For request entity-body headers (content), you need to modify the `HttpRequestMessage.Content.Headers1 collection. You are getting an exception telling you that basically.

So, you need code like this:

var content = new StringContent(...);
content.Headers.ContentType = ...;

You can also use methods like this as well:

var content = new StringContent(...);
content.Headers.Remove(...);
content.Headers.Add(...);

[guardrex]

@davidsh Ah ... ok ... thx ... that got the header squared away ...

stringContent.Headers.ContentType = new MediaTypeHeaderValue(contentType);

And 🎉 YES! 🎉 ... that cleared up the problem with ATS REST API: SharedKey works now! 😄 👍 🍻

My issue is solved, but do you want to leave this open given what @JonHanna reported?

[davidsh]

My issue is solved, but do you want to leave this open given what @JonHanna reported?

We will research this to determine if there are RFC issues. This code has existed for 5+ years and was previously reviewed by our RFC compliance experts. I'll leave this open for now.

[JonHanna]

I was incorrect.

While charset was originally defined only for use with text/* types, there's nothing to stop some particular type in another group using a copy of it, and RFC 3023 did exactly that with it strongly recommended to be used along with application/xml and suggested, and then RFC 4287 references that in turn in allowing charset with application/atom+xml. So it's not in the slightest invalid to have that parameter with that content type, sorry for my misleading comment.

[guardrex]

My concern about it remains: If the dev sets a value explicitly on new StringContent(), they will not be aware that charset is being added.

In any case where the header is part of a service-generated hash that will be checked against a supplied signature (without the charset, since the dev didn't know it was added to the request header), as is the case with Azure Storage REST Service, ... 💥 ... and lost time having to check a lot of little things partially because the Azure REST docs and exception messages are so poor. All the service comes back with is 'your auth header is hosed! ... check the signature.'

If the dev sets an explicit value, seems like it should be honored. At least mod the description to note that charset will be added automatically. It's possible that if I saw that pop-up in VS when I was laying down the code, it might have triggered more immediate interest in trying MediaTypeHeaderValue().

[guardrex]

Just for completeness, I checked the Azure Service REST Service using the default ...

StringContent("<data>", Encoding.UTF8, "application/atom+xml");

... but supplying the content type to the signature string as ...

string contentType = "application/atom+xml; charset=utf-8";
string stringToSign = 
    $"{requestMethod}\n\n{contentType}\n{dateInRfc1123Format}\n{canonicalizedResource}";

That works. It's still awfully cryptic tho without a little help.

[darrelmiller]

@JonHanna Usually it is the media type registration document that defines what parameters are allowed. In the application/atom+xml the registration refers explicitly back RFC3023 for the rules, as you stated.

Interestingly, the .net stack loves adding charset onto application/json but in that case charset is not defined by the media type registration. In fact RFC 7158 goes on to say,

Note: No "charset" parameter is defined for this registration.
Adding one really has no effect on compliant recipients.

It would be awesome if this could be fixed.

[davidsh]

We plan to fix this in CoreFx. We need to study if porting this to .NET Framework (Desktop) will have any app-compat issues.

[davidsh]

This will require a new overload to the StringContent constructor so that the charset isn't specified by default on the Content-Type request header. So, we'll need an API proposal for that.