[Windows] Got "OverflowError: Python int too large to convert to C long" on large non-exist PIDs

[XuehaiPan]

Summary

• OS: Windows
• Architecture: 64bit
• Psutil version: 5.9.5
• Python version: 3.11.3
• Type: core

Description

I'm facing a third-party bug (NVIDIA NVML Python bindings) that returns non-exist PIDs for GPU processes. The PID overflows the C long type limit and raises an OverflowError.

It would be nice if psutil can catch that error and reraise a new exception in a subtype of psutil.Error. For example, if the user passed an extremely large number to psutil.Process(pid), raise a psutil.NoSuchProcess error rather than OverflowError.

In [1]: import pynvml

In [2]: pynvml.nvmlInit()

In [3]: handle = pynvml.nvmlDeviceGetHandleByIndex(0)

In [4]: [p.pid for p in pynvml.nvmlDeviceGetComputeRunningProcesses(handle)]
Out[4]:
[1184,
 0,
 4294967295,
 4294967295,
 16040,
 0,
 4294967295,
 4294967295,
 19984,
 0,
 4294967295,
 4294967295,
 20884,
 0,
 4294967295,
 4294967295,
 26308,
 0,
 4294967295,
 4294967295,
 16336,
 0,
 4294967295,
 4294967295,
 5368,
 0,
 4294967295,
 4294967295,
 19828,
 0,
 4294967295]

Reproduce script:

import psutil

process = psutil.Process(0xFFFFFFFF)

On Windows, LONG_MAX = 0x7FFFFFFF (Microsoft C Limits on Integer Constants).

Traceback:

│ C:\Tools\Python3\Lib\site-packages\psutil\__init__.py:361 in _init                               │
│                                                                                                  │
│    358 │   │   self._exitcode = _SENTINEL                                                        │
│    359 │   │   # cache creation time for later use in is_running() method                        │
│    360 │   │   try:                                                                              │
│ ❱  361 │   │   │   self.create_time()                                                            │
│    362 │   │   except AccessDenied:                                                              │
│    363 │   │   │   # We should never get here as AFAIK we're able to get                         │
│    364 │   │   │   # process creation time on all platforms even as a                            │
│                                                                                                  │
│ ╭──────────────────────────────── locals ────────────────────────────────╮                       │
│ │ _ignore_nsp = True                                                     │                       │
│ │         pid = 4294967295                                               │                       │
│ │        self = <repr-error 'Python int too large to convert to C long'> │                       │
│ ╰────────────────────────────────────────────────────────────────────────╯                       │
│                                                                                                  │
│ C:\Tools\Python3\Lib\site-packages\psutil\__init__.py:719 in create_time                         │
│                                                                                                  │
│    716 │   │   The return value is cached after first call.                                      │
│    717 │   │   """                                                                               │
│    718 │   │   if self._create_time is None:                                                     │
│ ❱  719 │   │   │   self._create_time = self._proc.create_time()                                  │
│    720 │   │   return self._create_time                                                          │
│    721 │                                                                                         │
│    722 │   def cwd(self):                                                                        │
│                                                                                                  │
│ ╭──────────────────────────── locals ─────────────────────────────╮                              │
│ │ self = <repr-error 'Python int too large to convert to C long'> │                              │
│ ╰─────────────────────────────────────────────────────────────────╯                              │
│                                                                                                  │
│ C:\Tools\Python3\Lib\site-packages\psutil\_pswindows.py:694 in wrapper                           │
│                                                                                                  │
│    691 │   @functools.wraps(fun)                                                                 │
│    692 │   def wrapper(self, *args, **kwargs):                                                   │
│    693 │   │   try:                                                                              │
│ ❱  694 │   │   │   return fun(self, *args, **kwargs)                                             │
│    695 │   │   except OSError as err:                                                            │
│    696 │   │   │   raise convert_oserror(err, pid=self.pid, name=self._name)                     │
│    697 │   return wrapper                                                                        │
│                                                                                                  │
│ ╭───────────────────────────── locals ──────────────────────────────╮                            │
│ │   args = ()                                                       │                            │
│ │    fun = <function Process.create_time at 0x000002795AA671A0>     │                            │
│ │ kwargs = {}                                                       │                            │
│ │   self = <psutil._pswindows.Process object at 0x000002795CECF780> │                            │
│ ╰───────────────────────────────────────────────────────────────────╯                            │
│                                                                                                  │
│ C:\Tools\Python3\Lib\site-packages\psutil\_pswindows.py:948 in create_time                       │
│                                                                                                  │
│    945 │   │   # Note: proc_times() not put under oneshot() 'cause create_time()                 │
│    946 │   │   # is already cached by the main Process class.                                    │
│    947 │   │   try:                                                                              │
│ ❱  948 │   │   │   user, system, created = cext.proc_times(self.pid)                             │
│    949 │   │   │   return created                                                                │
│    950 │   │   except OSError as err:                                                            │
│    951 │   │   │   if is_permission_err(err):                                                    │
│                                                                                                  │
│ ╭──────────────────────────── locals ─────────────────────────────╮                              │
│ │ self = <psutil._pswindows.Process object at 0x000002795CECF780> │                              │
│ ╰─────────────────────────────────────────────────────────────────╯                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

[giampaolo]

Mmm. Yes, I think it makes sense. The error originates from PyArg_ParseTuple (used pretty much everywhere). The Process class can invoke PyArg_ParseTuple on __init__ (hence only once) and turn OverflowError into NoSuchProcess.

The reason why I think this is a good idea is because OverflowError is only raised when invoking C functions. On Linux most of Process APIs are implemented in python instead, by reading /proc, so we do not get OverflowError in those cases (we get NoSuchProcess). As such, making this change would make things more consistent across all platforms, which will always rase NoSuchProcess.

[XuehaiPan]

I think it makes sense. The error originates from PyArg_ParseTuple (used pretty much everywhere). The Process class can invoke PyArg_ParseTuple on init (hence only once) and turn OverflowError into NoSuchProcess.

Could we capsule and expose PyArg_ParseTuple(args, _Py_PARSE_PID, &pid) as a function to cext? Then we can call it in Process._init before any cext call.

[giampaolo]

Yeah, more or less that's what I was thinking.