Arm64: Environment.ProcessorCount returns wrong value on higher core machine

[kunalspathak]

In #45943, we did a breaking change to have Environment.ProcessorCount take processor affinity into account on Windows. It works well on x64 machines, but on arm64 machines with higher CPU count, it returns wrong value.

Although #47427, calls out that the code that relies on Environment.ProcessorCount for parallelism, should update it to scale better because of reduced return value. However, there is performance sensitive code in IOThreadPool that relies on Environment.Processor to decide on the number of "IO completion poller threads" to use in the application.

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.IO.Windows.cs

Lines 41 to 43 in 32320b1

	int processorsPerPoller =
	AppContextConfigHelper.GetInt32Config("System.Threading.ThreadPool.ProcessorsPerIOPollerThread", 12, false);
	return (Environment.ProcessorCount - 1) / processorsPerPoller + 1;

On higher core Arm64 machine, had ProcessorCount returned correct value, I could see that we would create more IO completion threads to handle more concurrent requests. In an internal application, I can see it boosts performance by 2.25X on higher core Arm64 machines.

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

In #45943, we did a breaking change to have Environment.ProcessorCount take processor affinity into account on Windows. It works well on x64 machines, but on arm64 machines with higher CPU count, it returns wrong value.

Although #47427, calls out that the code that relies on Environment.ProcessorCount for parallelism, should update it to scale better because of reduced return value. However, there is performance sensitive code in IOThreadPool that relies on Environment.Processor to decide on the number of "IO completion poller threads" to use in the application.

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.IO.Windows.cs

Lines 41 to 43 in 32320b1

	int processorsPerPoller =
	AppContextConfigHelper.GetInt32Config("System.Threading.ThreadPool.ProcessorsPerIOPollerThread", 12, false);
	return (Environment.ProcessorCount - 1) / processorsPerPoller + 1;

On higher core Arm64 machine, had ProcessorCount returned correct value, I could see that we would create more IO completion threads to handle more concurrent requests. In an internal application, I can see it boosts performance by 2.25X on higher core Arm64 machines.

Author:	kunalspathak
Assignees:	-
Labels:	area-System.Threading, untriaged
Milestone:	-

[kunalspathak]

@kouvel @mangod9

[mangod9]

this looks to be hardware specific -- I couldnt repro on an arm64 VM with 32 cores.

[kunalspathak]

When I set COMPlus_GCCpuGroup=1 and COMPlus_Thread_UseAllCpuGroups=1 , it reports current number of cores.

[AntonLapounov]

I am helping @kunalspathak to debug this.

[AntonLapounov]

I can confirm the issue. The program reports the incorrect value of Environment.ProcessorCount every other run, which is certainly weird.

[mangod9]

hmm, interesting. Is this every other run of the process or subsequent calls in the same process? I dont seem to be able to repro it on a 32 core VM fyi.

[AntonLapounov]

The machine in question has two processor groups of different sizes. Unless you set environment variables to use all CPU groups, Environment.ProcessorCount returns the number of processors in the processor group the OS started the process on. Therefore, you may see different returned values from run to run and that is the expected behavior.

[mangod9]

ok that makes sense. Need to determine why there is an asymmetric config on that hardware.

[AntonLapounov]

According to documentation, in Windows 11 processes are no longer constrained to a single processor group by default. I guess that means we should consider UseAllCpuGroups enabled by default, which would allow to avoid this issue.

[kunalspathak]

@AntonLapounov - any further update on this?

[AntonLapounov]

I have started discussion #67308 and an internal email thread with OS folks regarding this. I assume you can set the environment variables as a workaround until we have a fix.

[janvorli]

@AntonLapounov can you please set the milestone of this issue to remove the untriaged label?

[mangod9]

@AntonLapounov @kunalspathak guess we dont plan on fixing for 7 right?

[AntonLapounov]

Yes, apps will have to opt-in to make all processor groups available to the runtime.

[kunalspathak]

Do we plan to include the documentation of the environment variable to use all processor groups for Windows 11/Windows server 2022?

[AntonLapounov]

Those configuration settings were introduced a long time ago and are already documented. If you think the documentation should be improved, you may submit a documentation PR.

[kunalspathak]

If you think the documentation should be improved

They need to be certainly improved because of change in behavior on Win11 and with many-core machines are getting more common. Under what API/category should the documentation go to?

[jkotas]

Under what API/category should the documentation go to?

Affected APIs should get a note:

• https://docs.microsoft.com/en-us/dotnet/core/runtime-config/threading#cpu-groups
• https://docs.microsoft.com/en-us/dotnet/api/system.environment.processorcount
• https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.process.processoraffinity

I suspect that we may still end up doing more product changes here given the feedback in #72441. I would wait for that issue to get resolved before starting on documentation edits.

(If you have access to a Win2022 machine with large number of cores, it would be useful to get answer to #72441 (comment) .)