HttpRequestMessage does not respect TryAddWithoutValidation when it deals with User Agent.

[emorell96]

Description

I am using .Net 5 right now, and I am trying to send a request to an API.

I am setting the User Agent using the TryAddWithoutValidation method since I need to use a non compliant User Agent like "My App / 1.0.5" (with the spaces).

Yet when I check the request on Fiddler I see that .Net removes my spaces between App and / and / and 1.0.5.

var requestMessage = new HttpRequestMessage(HttpMethod.Get, "https://test.com");
requestMessage.Headers.TryAddWithoutValidation("User-Agent", "My Android App / 1.0.5");

And you will see in Fiddler it appears as User-Agent: "My Android App/1.0.5" (without the spaces) causing a 500 error in my case.

I thought the whole idea of TryAddWithoutValidation was that it would leave my string values alone and just send what I told it to send.

Configuration

.net 5.0 in Windows 10 x64

Regression?

As far as I know this seems to be a recurrent issue in .net core: #35335
#21131

Other information

I have tried using ParseAdd of the UserAgent, and using the default headers of the HttpClient object, all the same result. My user agent gets parsed and transformed. Any help would be appreciated since I cannot communicate with the API unless this gets fixed.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I am using .Net 5 right now, and I am trying to send a request to an API.

I am setting the User Agent using the TryAddWithoutValidation method since I need to use a non compliant User Agent like "My App / 1.0.5" (with the spaces).

Yet when I check the request on Fiddler I see that .Net removes my spaces between App and / and / and 1.0.5.

requestMessage.Headers.TryAddWithoutValidation("User-Agent", "My Android App / 1.0.5");

And you will see in Fiddler it appears as User-Agent: "My Android App/1.0.5" (without the spaces) causing a 500 error in my case.

I thought the whole idea of TryAddWithoutValidation was that it would leave my string values alone and just send what I told it to send.

Configuration

.net 5.0 in Windows 10 x64

Regression?

As far as I know this seems to be a recurrent issue in .net core: #35335
#21131

Other information

I have tried using ParseAdd of the UserAgent, and using the default headers of the HttpClient object, all the same result. My user agent gets parsed and transformed. Any help would be appreciated since I cannot communicate with the API unless this gets fixed.

Author:	emorell96
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[ManickaP]

Triage: we should at least try to investigate for 6.0.

[emorell96]

I posted this in StackOverflow and someone else experienced it as well: SO
I investigated a little and it seems like the .net still parses the user agent and when it sees "App / 1.0.22" it translates that into a ProductHeaderValue object with Name = "App" and Version ="1.022" without spaces.

[geoffkizer]

Fiddler is acting as a proxy here, right?

There's some code to propagate the User-Agent header on proxy tunnel creation here: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs#L1460

That code is causing the User-Agent to get parsed, which means we will reformat it when we send it. We should change the code to not force parsing of User-Agent here.

That said, if you are not using a proxy (or not using HTTPS) then this shouldn't happen.

[emorell96]

Fiddler is a proxy to check the requests but it is not set on code but on Windows. I don't know if that would change anything. All I can say is that I know the user agent gets parsed because the API blocks my calls, and I viewed the requests through Fiddler to confirm this. You are welcome to investigate this:

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.TryAddWithoutValidation("user-agent",    "my android app / 5.23.0");

var url = "https://google.com";
var requestMessage = new HttpRequestMessage(HttpMethod.Get, url);
var response = await httpClient.SendAsync(requestMessage);

and see what happens.

[geoffkizer]

If you are using DefaultRequestHeaders to set the header value then you'll also hit this issue: #49463

[karelz]

Fixed by #49673

I verified that it works fine on latest 6.0 daily build.