(Windows7-64bits) get_memory_info fails when called toooo many times

[giampaolo]

From [email protected] on December 08, 2011 17:55:04

It seams that there is a limit to the number of times that we can call 
get_memory_info for a process. This causes problems when I need to
monitor the memory usage of a very long process. 

Run the attached file on a windows 7 64-bit machine (python 2.7, psutil 0.4.0) 
to replicate the issue. The program starts an infinate loop that asks for the 
current process memory. If left alone it should run forever but it fails with:

WindowsError: [Error 1450] Insufficient system resources exist to complete the 
requested service

The problem happens each time when the function has been called ~16711544 times. 

Thanks,
Ioannis

Attachment: issue.py

Original issue: http://code.google.com/p/psutil/issues/detail?id=236

[giampaolo]

From [email protected] on December 08, 2011 09:11:38

That sounds like it may be a memory leak issue. Are you able to run the 
test_memory_leaks.py script on your system? (It's in psutil test/ directory if 
you download the source).

Labels: OpSys-Windows7 64bit

[giampaolo]

From [email protected] on December 08, 2011 13:16:47

Running the memory leak test results in three failures, but not at the 
get_memory_info (see traceback). I think that the problem is that every time
a new process handle is opened in _psutil_mswindows.c. Maybe this should be 
done once at the object creation. Furthermore, microsoft suggests that the 
process handle has been opened with PROCESS_QUERY_INFORMATION | PROCESS_VM_READ
( 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682050(v=vs.85).aspx) 
this might be also something to look at.

--------------
Test Traceback
--------------

test_cmdline (__main__.TestProcessObjectLeaks) ... ok
test_create_time (__main__.TestProcessObjectLeaks) ... ok
test_get_connections (__main__.TestProcessObjectLeaks) ... FAIL
test_get_cpu_times (__main__.TestProcessObjectLeaks) ... ok
test_get_memory_info (__main__.TestProcessObjectLeaks) ... ok
test_get_num_threads (__main__.TestProcessObjectLeaks) ... ok
test_get_open_files (__main__.TestProcessObjectLeaks) ... ok
test_get_threads (__main__.TestProcessObjectLeaks) ... ok
test_getcwd (__main__.TestProcessObjectLeaks) ... ok
test_gids (__main__.TestProcessObjectLeaks) ... skipped-ok
test_is_running (__main__.TestProcessObjectLeaks) ... ok
test_name (__main__.TestProcessObjectLeaks) ... ok
test_ppid (__main__.TestProcessObjectLeaks) ... ok
test_resume (__main__.TestProcessObjectLeaks) ... ok
test_status (__main__.TestProcessObjectLeaks) ... ok
test_terminal (__main__.TestProcessObjectLeaks) ... skipped-ok
test_uids (__main__.TestProcessObjectLeaks) ... skipped-ok
test_username (__main__.TestProcessObjectLeaks) ... FAIL
test_cpu_times (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_io_counters (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_partitions (__main__.TestModuleFunctionsLeaks) ... ok
test_disk_usage (__main__.TestModuleFunctionsLeaks) ... ok
test_get_pid_list (__main__.TestModuleFunctionsLeaks) ... ok
test_network_io_counters (__main__.TestModuleFunctionsLeaks) ... FAIL
test_per_cpu_times (__main__.TestModuleFunctionsLeaks) ... ok
test_phymem_usage (__main__.TestModuleFunctionsLeaks) ... ok
test_pid_exists (__main__.TestModuleFunctionsLeaks) ... ok
test_process_iter (__main__.TestModuleFunctionsLeaks) ... ok
test_virtmem_usage (__main__.TestModuleFunctionsLeaks) ... ok

======================================================================
FAIL: test_get_connections (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ionnis\Projects\psutil-0.4.0\test\test_psutil.py", line 150, in inner
    return fun(self, *args, **kwargs)
  File ".\test_memory_leaks.py", line 158, in test_get_connections
    self.execute('get_connections')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=15777792, rss3=16179200, difference=401408

======================================================================
FAIL: test_username (__main__.TestProcessObjectLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\ionnis\Projects\psutil-0.4.0\test\test_psutil.py", line 150, in inner
    return fun(self, *args, **kwargs)
  File ".\test_memory_leaks.py", line 120, in test_username
    self.execute('username')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=17195008, rss3=17219584, difference=24576

======================================================================
FAIL: test_network_io_counters (__main__.TestModuleFunctionsLeaks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".\test_memory_leaks.py", line 202, in test_network_io_counters
    self.execute('network_io_counters')
  File ".\test_memory_leaks.py", line 73, in execute
    % (rss2, rss3, difference))
AssertionError: rss2=17641472, rss3=17666048, difference=24576

----------------------------------------------------------------------
Ran 29 tests in 466.125s

FAILED (failures=3)

[giampaolo]

From [email protected] on December 08, 2011 13:26:18

Does the memory information get returned properly? 

I can't see any obvious reason a handle leak would be occurring. We close the 
handle on all code paths in the memory function itself.

[giampaolo]

From [email protected] on December 08, 2011 14:23:02

I cannot figure it out either, I have tried using pywin32 and it has no problem going
upto 2**25 call times. However, I am using it with PROCESS_QUERY_INFORMATION | 
PROCESS_VM_READ.

[giampaolo]

From [email protected] on December 08, 2011 14:47:32

Well, I ran the test script for over an hour on my system and can't get it to 
error out. However, I'm not on Windows 7 64bit so that may be a factor. 
Giampaolo might be able to test, I think he has a Win 7 system. 

The flags used when the handle is opened wouldn't have an effect on this issue, 
but in any case, the handle used by psutil for get_memory_info() is already 
being opened with PROCESS_QUERY_INFORMATION | PROCESS_VM_READ 
(handle_from_pid() function in psutil/arch/mswindows/process_info.c) 

There shouldn't be any functional difference but is it possible for you to try 
with the latest SVN version just in case?

[giampaolo]

From [email protected] on December 08, 2011 15:34:06

Looking at the SVN version I can see that there is an additional CloseHandle call
which is not present in the 0.4.0 source. I will try to compile the updated 
source, but I think that it should work fine.

[giampaolo]

From [email protected] on December 08, 2011 16:02:24

Hrm, my fault then - I thought that change had made it into 0.4.0 but 
apparently not. That would certainly account for the issue if there's a handle 
leak since the error code indicates running out of file descriptors/handles.

[giampaolo]

From g.rodola on December 09, 2011 11:41:35

There was an unclosed handle indeed, fixed in r1219 by David who didn't file an 
issue or updated HISTORY.

Status: FixedInSVN
Labels: Milestone-0.4.1

[giampaolo]

From g.rodola on December 14, 2011 15:17:58

Labels: -Priority-Medium -OpSys-Windows7 Priority-High OpSys-Windows

[giampaolo]

From g.rodola on December 14, 2011 15:18:36

Status: Fixed

[giampaolo]

From g.rodola on December 14, 2011 15:51:42

This is now fixed in 0.4.1 version.

[giampaolo]

From g.rodola on March 02, 2013 04:06:03

Updated csets after the SVN -> Mercurial migration: r1219 == revision 
d6716872097a