HttpClient.DefaultProxy should respect IE proxy changes

[ahdung]

Env: Windows server 2008 R2 SP1

In .netfx, this API work fine, if turn on Fiddler capturing, it capturing.
But in .netcore (3.1+), this API always return HttpNoProxy, no longer respect the IE proxy settings.

How get the IE proxy settings? Please help.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Env: Windows server 2008 R2 SP1

In .netfx, this API work fine, if turn on Fiddler capturing, it capturing.
But in .netcore (3.1+), this API always return HttpNoProxy, no longer respect the IE proxy settings.

How get the IE proxy settings? Please help.

Author:	ahdung
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[ahdung]

I think this is really a mistake:
https://source.dot.net/#System.Net.Requests/System/Net/WebRequest.cs,564

GetSystemWebProxy acquire from HttpClient.DefaultProxy, but the latter one is a settable Property, How can do this? the GetSystemWebProxy API should respect its name meaning.

[karelz]

HttpClient.DefaultProxy uses IE proxy, unless it is overridden by environment variables, or by the app itself (it is intentionally settable): https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/SystemProxyInfo.Windows.cs

I am curious why that detection does not work on your machine. Are you able to enable logging in your environment, or even better, debug into the proxy initialization?

WebRequest.GetSystemWebProxy() intentionally uses HttpClient.DefaultProxy (see PR dotnet/corefx#41692) and allows application to overdrive/customize system proxy. It is intended for better flexibility and to have common proxy value in the application.
BTW: WebRequest class is obsolete and HttpClient is recommended to be used instead -- see docs

[ahdung]

Thanks for your response.
I'm not talking the HttpClient.DefaultProxy can't be settable, i'm talking WebRequest.GetSystemWebProxy() shouldn't use it, this API should always get the IE proxy settings, like netfx does. Think, if i set DefaultProxy to a custom proxy, how can call it system web proxy? unless the system web proxy not equals the IE proxy in .netcore context, but this means that an API with the same name has completely different meanings and behaviors in netfx and net[core], this is a horrible thing for migrate.

The another problem is the WebRequest.GetSystemWebProxy() is cached, when it gets the IE proxy settings at first time, it never change, but you should know, the IE proxy settings is a dynamic thing. look, in netfx, i use Fiddler capture the web requests, when i start capturing, it capturing, when i stop it, it stop, everything is controllable and expected. But now, if i start program with the capture turned off, it will never be able to capture, conversely, when i start the program with the capture turned on, it will always capture and cannot be stopped unless close program or fiddler. Basically, you did something like this:

// the right way:
DateTime GetNowTime() => DateTime.Now;

// your way
DateTime _now = DateTime.Now;
DateTime GetNowTime() => _now;

Finnally, i don't want to expand discussion of WebRequest and HttpClient, i just care How to get the IE proxy settings in real time? Thank you.

[karelz]

We do not have ability to get current IE settings in .NET Core.

.NET Framework watches for registry changes that signal change in proxy settings (reg key HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections - see AutoWebProxyScriptEngine.ListenForRegistry), then query WinHttp in WinHttpWebProxyBuilder

On .NET Core we read proxy settings only once at first use, by querying WinHttp in WinInetProxyHelper - it is the same function that .NET Framework uses.

I assume, we could implement similar registry change monitoring and refresh proxy info from WinHttp in such case (assuming it was not overridden by app/env variables).
We should check that it is officially supported way from WinHttp. And probably think about impersonation impact (something .NET Framework does).
We would also need to figure out how to update proxy in HttpConnectionPoolManager - currently it is designed to take proxy snapshot at the creation time.

The key remaining question is: How much does it matter? Is it only developer scenario (Fiddler turned on and off during application execution), and how commonly is it needed? Or are there other scenarios that would benefit from this?

[ahdung]

This is really regrettable. Ofcourse not only developer scenario, many programs with network access capabilities usually provide an option called Use IE proxy or Use System proxy, and .net framework has also supported this since 2.0, now you tell me the great .NET can't do this.
Even if we do not discuss whether it can be implemented, a Property(DefaultProxy) that can be arbitrarily sets cannot be called system proxy, you really did wrong on this thing, why don't you just obselete this API?
Apart from being alive, many things are not much matter, but about experience.

[karelz]

@ahdung I am sorry that you ran into this the hard way. Yes, it is some incompatibility with .NET Framework, which hit you by surprise. We wish we could be 100% compatible, but you can probably imagine that it is impossible task and we have to pick our battles and choose where to invest.

To answer your concerns:

The obsoletion of WebRequest didn't make it into 5.0 (which is sad), but is at least in 6.0 (master) branch, with some follow ups needed: #41759

Regarding DefaultProxy -- it is up to the application, or the environment to override system proxy - aka lie to itself, or bring even better system proxy detection or other proxy detection than .NET Core does.
If there are real-world cases, where the overriding causes problems, we'd love to hear about them. Let's not dismiss the API usefulness just on principle.

To make further progress, there are several separate "issues" raised here:

1. WebRequest.GetSystemWebProxy can be technically overridden by application/environment, which may be unexpected.
   • Open question: Does it cause real-world problems to applications?
2. Inability to get true/unmodified system (IE) proxy settings in .NET Core.
   • Question: How common is this scenario? Is it something that could be solved in external nuget package potentially with better detection on each OS? (at least as temporary workaround)
3. DefaultProxy does not react to changes to OS settings after it was initialized.
   • Question: How important is this scenario? Does it impact only dev scenarios, or are dynamic proxy changes also real-world scenarios for end users (non-developers)?
   • Note: This is IMO the most important question as the answer fundamentally impacts our design and assumptions we made in our new HTTP stack.

@ahdung did I miss any angle? From the list, which of the problems do you think is worth solving in .NET Core?

[ahdung]

@karelz My English is not good, I try to understand what you mean, please point out if there is any misunderstanding.

Regarding WebRequest, i hope don't obsolete it, although it has some issues now. i believe that many old apps still dependent it, this is absolutely a major breaking change, please be cautious.

Regarding DefaultProxy, it can be changed, that is not a problem in my opinion, like WebRequest.DefaultWebProxy, it not called system something. but it is precisely because it can be changed so i think should not be let GetSystemWebProxy use it. DefaultProxy can use GetSystemWebProxy on init, but GetSystemWebProxy shouldn't use DefaultProxy never.

Regarding "issues":

1. The IE proxy is inherently can be modified, it is a dynamic thing, everyone knows that, it's not unexpectedly.
2. I hope you sir support this need, even if you don't want use the old GetSystemWebProxy, maybe you can provide a new API called GetIEProxy something, otherwise the developers can only access registry for help self. use nuget is ok for me.
3. I hope DefaultProxy like WebRequest.DefaultWebProxy, if no set, it always follow the system proxy changing. Allowing end users to select proxy mode and set up his custom proxy has always been a feature pursued by network software providers, this is a about user's freedom thing. Think, a user sets his global proxy(IE proxy), and all his network software are work follow this, all thing are perfect he feel, but, there is a software which is developed by .net[core] that does not follow it, and can only set it separately, then he will hate this software. As we know, .net[core] not only about web, the Client-end softwares should also be taken care of.

[karelz]

My English is not good, I try to understand what you mean, please point out if there is any misunderstanding.

Absolutely. Happy to help. Let me know if some of my explanations are not clear.

Regarding WebRequest, i hope don't obsolete it, although it has some issues now. i believe that many old apps still dependent it, this is absolutely a major breaking change, please be cautious.

It is already obsoleted. Old apps can use it. They will not get full functionality or high performance though.

i think should not be let GetSystemWebProxy use it. ... GetSystemWebProxy shouldn't use DefaultProxy never.

I understand what you're saying. Given that GetSystemWebProxy is obsoleted API, we will keep it as it is.

Regarding your answers to "issues" 1-3: I am looking for real world scenarios. Cases you have hit where it matters. How important are those scenarios / cases and how common are they?
I agree with you that problems can happen. However, rarely - IE proxy changes are rare. Therefore, I suggest to keep it as is and focus on more important bugs, features. Based on the information so far, it does not seem like common scenario that would hit too many users.
Does that make sense?

[karelz]

Triage: We are ok to take a change with monitoring registry for IE proxy change and try to apply it. It is low pri for us though.
It may impact binary size for ILLinker scenario ...

[ahdung]

@karelz I must say that this is a great news for me, high expectations, thank you for your understanding.
But there seems have a little issue here, that if i changed HttpClient.DefaultProxy, how do i re-get the IE proxy settings again? So I think it’s better to provide another API?

[karelz]

@ahdung we do not plan to support direct access to IE proxy via new API. If part of your app decides to override it (incl. env. variables), then it is what will be set.
You can try to take a snapshot early on, before anything in your app overrides it.

[ahdung]

@karelz I see.

[Eli-Black-Work]

I think we're running into a similar issue. I'd like to explain our use case 🙂

When our company laptops are on the company network, they use a proxy. When they are off the company network, we're required to use our company VPN. However, now that people have started working from home, some people are doing this:

1. Use the VPN to join meetings. (This means that they're using the proxy)
2. When not in a meeting, don't use the VPN and disable the proxy (via Start Menu -> Proxy Settings), because the VPN is slow.

Disabling the VPN/proxy is against company rules, but it seems that people are doing it, and HttpClient doesn't seem to support this.

Our code looks as follows:

Startup.cs

services
    .AddHttpClient<IExample, Example>(httpClient =>
    {
        httpClient.BaseAddress = new Uri(Configuration["..."], UriKind.Absolute);
    })
    .ConfigurePrimaryHttpMessageHandler(sp =>
         {
             return new HttpClientHandler()
             {
                 UseProxy = true,
                 DefaultProxyCredentials = CredentialCache.DefaultCredentials
             }
         });

Example.cs

public class Example
{
    public Example(HttpClient httpClient)
    {
        _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
    }

    public async Task Test()
    {
        await _httpClient.GetAsync("http://example.com");
    }
}

When I do the following:

1. Log on to the VPN (so we're using the company proxy) and run the app. Works fine.
2. Keep the app running. Log off of the VPN, disable the proxy (via Start Menu -> Proxy Settings), and try doing Example::Test() again. This results in the following exception:

System.Net.Http.HttpRequestException: 'The proxy tunnel request to proxy '<_proxy_url_redacted_by_me_>' failed with status code '400'."'

If I log off the VPN, disable the proxy, and then start the app, everything works fine.

Not sure if this use case is worth supporting, but thought I'd mention it 🙂 I think a more common use case might be that the company updates the proxy's .PAC file while the app is running.

[wfurt]

this make sense to me. I'm wondering if we could do refresh of proxy setting when we fail to connect to proxy. any thoughts on this @geoffkizer? (we already have some re-try mechanism inside of the Windows proxy, we would just need mechanism to kick another lookup)

[wfurt]

Actually looking at #63788, since we already hook for NetworkChange, that may work as well as trigger to re-check proxy setting.

[lindexi]

@karelz Thank you. This behavior is like a bug. The .NET Core application can not handle the system web proxy changed.

The step:

1. Open the Fiddler
2. Open the .NET Core application and use the HttpClient to get something
3. Close the Fiddler

And then we will find the .NET Core application can not connect the network.

[ManickaP]

Closing as a dupe of #95252, see #95252 (comment)

Optimistically planned for 9.0.