Cannot get_cpu_percent() when run as Admin on Win7

[giampaolo]

From [email protected] on May 09, 2011 12:10:47

What steps will reproduce the problem?  
1. Create ProcWrapper.py as a class and put the following into the class as a method

def get_current_processes(self):
        processes = []
        for process in psutil.process_iter():
            try:
                p_cpu_perc = str(process.get_cpu_percent())
                p_mem_vms = str(process.get_memory_info().vms) 
                p_mem_rss = str(process.get_memory_info().rss) 
                p_num_threads = str(process.get_num_threads())

                processes.append({'pid': process.pid, 
                                  'name': process.name, 
                                  'status': str(process.status), 
                                  'cpu_perc': p_cpu_perc, 
                                  'memory_vms': p_mem_vms, 
                                  'memory_rss': p_mem_rss, 
                                  'num_threads': p_num_threads})
            except:
                processes.append({'pid': process.pid, 
                                  'name': process.name, 
                                  'status': str(process.status), 
                                  'cpu_perc': '-1', 
                                  'memory_vms': '-1', 
                                  'memory_rss': '-1', 
                                  'num_threads': '-1'})
        return processes

2. Create driver.py script that instantiate the class and calls the 
get_current_processes() method. Store the returned results in a variable.

3. Execute cmd as Administrator.

4. Run "python driver.py" 

What is the expected output?  


What do you see instead?  
Expected: A list of all the processes running without processes with '-1' value.

Actual: A list of all the processes running with processes with '-1' value. 

What version of psutil are you using? What Python version?  
Version: 0.2.1
Python: 2.6 

On what operating system? Is it 32bit or 64bit version?  
Win7 64-bit 

Please provide any additional information below.  
Tried from "python" prompt as administrator, it works. However, when running as 
a script, it failed.

Original issue: http://code.google.com/p/psutil/issues/detail?id=161

[giampaolo]

From [email protected] on May 09, 2011 07:02:08

Can you try running this without catching all the exceptions, or at least print 
a stack trace for a few of them? The code as written is swallowing all types of 
exceptions and falling through to the same error case so it's pretty much 
impossible to tell what may be going wrong. 

Thanks, 

-Jay

Labels: OpSys-Windows7 64bit

[giampaolo]

From g.rodola on May 09, 2011 07:20:09

Also, the process.get_cpu_percent() in there causes a big slowdown since by 
default it has a timeout of 0.1 seconds (hence, say you have 100 processes 
you'll wait 10 seconds before that code block returns).

If you want to get an accurate CPU percent usage you'll have to iterate over 
all processes twice, by setting an interval of 0:

>>> p.cpu_percent(interval=0)  # this returns immediately
0.0
>>> # wait some time (e.g. by iterating over other processes)
>>> p.cpu_percent(interval=0)
11.3  # this is meaningful

[giampaolo]

From [email protected] on May 09, 2011 19:50:16

thanks for the speedy response. i have changed the code to

def get_current_processes(self):
        processes = []
        for process in psutil.process_iter():
            try:
                p_cpu_perc = str(process.get_cpu_percent(interval=0))
                p_mem_vms = str(process.get_memory_info().vms) 
                p_mem_rss = str(process.get_memory_info().rss) 
                p_num_threads = str(process.get_num_threads())

                p = {'pid': process.pid, 
                     'name': process.name, 
                     'status': str(process.status), 
                     'cpu_perc': p_cpu_perc, 
                     'memory_vms': p_mem_vms, 
                     'memory_rss': p_mem_rss, 
                     'num_threads': p_num_threads, 
                     'err': ''}

                p_cpu_perc = str(process.get_cpu_percent(interval=0))
                p['cpu_perc'] = p_cpu_perc

                processes.append(p)
            except Exception as ex:
                print type(ex)
                processes.append({'pid': process.pid, 
                                  'name': process.name, 
                                  'status': str(process.status), 
                                  'cpu_perc': '-1', 
                                  'memory_vms': '-1', 
                                  'memory_rss': '-1', 
                                  'num_threads': '-1', 
                                  'err': '1'})

        return processes

1. it now calls cpu_percent() twice as suggested.

2. it now prints out the exception. the problem now, is if i do not use the 
type(ex) function, it will just print out 

(pid=<pid>)

When I use type(ex), it prints out

<class 'psutil.error.AccessDenied'>

This is the same error when calling p.get_cpu_percent() without being an 
administrator. 

The only processes that is throwing this exception are

{'status': 'running', 'num_threads': '-1', 'name': 'System Idle Process', 
'err': '1', 'memory_vms': '-1', 'memory_rss': '-1', 'pid': 0,
 'cpu_perc': '-1'}
{'status': 'running', 'num_threads': '-1', 'name': 'System', 'err': '1', 
'memory_vms': '-1', 'memory_rss': '-1', 'pid': 4, 'cpu_perc':
'-1'}

[giampaolo]

From [email protected] on May 10, 2011 06:57:50

One problem is you're catching all exceptions during retrieval of the entire 
process information. For example, if get_num_threads() fails, then you're 
throwing away ALL the process information, even if memory, cpu, etc. are all 
collected successfully. 

I would really recommend re-tooling your code so that it only replaces the 
functions that actually throw an AccessDenied exception with a -1 value. I 
suspect you're running into AccessDenied for only one or two of the process 
functions for the two System-owned processes, but the way your code is designed 
it causes you to be unable to get any info for the process at all. It would 
also be helpful to know which function(s) specifically are throwing 
AccessDenied even as an Administrator user for the System processes.

-Jay

[giampaolo]

From [email protected] on May 10, 2011 20:53:10

I am a newbie in Python. So, I am not sure how to re-tool for it show more details. 

Anyway, I have excerpt the code and place it as a single script:

===
import psutil
import time

for process in psutil.process_iter():
    try:
        print 'pid: ' + str(process.pid)
        print 'name: ' + process.name

        # error begins here
        p_cpu_perc = process.get_cpu_percent(interval=0)
        time.sleep(1)
        p_cpu_perc = process.get_cpu_percent(interval=0)

        print 'cpu percent: ' + str(p_cpu_perc)
        print "===\n"
    except Exception, err:
        print "error"
        print str(type(err))
        print str(err)
        break
===

Instead of calling everything, I have made it stop when an exception occurs. 
When I run this script with the following steps:

1. click Start
2. type 'cmd' on the search bar. 
3. press Ctrl + Shift + Enter to run as Administrator
4. Go to the directory where the Python file (psutil_test.py) is. 
5. python psutil_test.py

And the following is printed on screen.

===
C:\Users\myaccount>python psutil_test.py
pid: 0
name: System Idle Process
cpu percent: 0.0
===

pid: 4
name: System
error
<class 'psutil.error.AccessDenied'>
(pid=4, name='System')

C:\Users\myaccount>
===

This is all I can get. This shows that the process 'System' has AccessDenied problem. 

I am not sure how to from here. If anyone here has an idea, please let me know. 

The same code works on WinXP with no problems. However, when I change from 
process.get_cpu_percent() to process.status, WinXP just crash with a crash 
dialog box. I will avoid using process.status for now. 

The same in Win7 also occurs when process.get_memory_info() is used. 

However, this problem goes away when they are entered in the Python prompt. It 
only happens on Win7 running from a Python scripted file. 

PS Please do not mind the time.sleep() there. This is just to show the problem. 
Performance is not a concern.

[giampaolo]

From [email protected] on May 13, 2011 19:31:46

> This is all I can get. This shows that the process 'System' has AccessDenied
> problem.
> 
> I am not sure how to from here. If anyone here has an idea, please let me
> know.

What I mean is, you're trying to read PID, name, status, CPU, mem and threads
all at once, when creating the dictionary. Since any of these individual calls
can throw an exception (AccessDenied, NoSuchProcess etc) that means that if
one fails, the entire dictionary for that process is created with "-1" values.

I'm not sure what you're trying to do with your code, but you're better off
wrapping each of these calls in a try/except block so that you can set the
"-1" error values only for those calls that fail due to an exception. Make
sense?

Depending on what you're doing though, this may be an unncessary step. The
process objects returned from process_iter() already return the values as
properties so you don't gain much by just putting the process objects in a
dictionary. Again, I'm not sure what you're doing with the dictionary so maybe
you have a good reason for doing it this way. 

> The same code works on WinXP with no problems. However, when I change from
> process.get_cpu_percent() to process.status, WinXP just crash with a crash
> dialog box. I will avoid using process.status for now.
>
> The same in Win7 also occurs when process.get_memory_info() is used.

This is the first I've heard of a crash in get_cpu_percent(), I'd like to see
if this is reproducible on XP here. Nothing should be crashing in psutil even
if it's an access issue, it should only throw an exception. A full on crash is
concerning if it's reproducible.

> However, this problem goes away when they are entered in the Python prompt. It
> only happens on Win7 running from a Python scripted file.

Are you talking about the crash, or the AccessDenied errors? I am not sure how
this would make a difference. If you run the script manually by invoking
python.exe with the script name does that make any difference, versus
double-clicking the .py file or however you're launching it now?

[giampaolo]

From [email protected] on May 17, 2011 18:22:37

Right. I have modified my codes. This time I do not use dictionary anymore. I 
write the results out directly to an XML file. The initial thought was to send 
the information to a database via POST. That's why I used dictionary to easily 
place things up. I do not have extensive Python background, but I do have PHP 
background. So, I thought a dictionary will make my life easier. 

Anyway, the updated codes as below:

===

import psutil
import time

log_file = "C:/temp/ps.log.xml"
handle = open(log_file, "w")
handle.write("<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n")
handle.write("<processes>\n")

for process in psutil.process_iter():
    p_cpu_perc = "-1.00"
    p_mem_vms = "-1"
    p_mem_rss = "-1"
    p_num_threads = "-1"
    p_status = "unknown"
    p_error = ""

    # get CPU perc 1st time
    try:
        p_cpu_perc = str(process.get_cpu_percent(interval=0))
    except Exception, err:
        p_error = p_error + "get_cpu_percent() [1st time]: " + str(err) + " : " 
+ str(type(err)) + "\n"

    # get VMS
    try:
        p_mem_vms = str(process.get_memory_info().vms) 
    except Exception, err:
        p_error = p_error + "get_memory_info().vms: " + str(err) + " : " + 
str(type(err)) + "\n"

    # get RSS
    try:
        p_mem_rss = str(process.get_memory_info().rss) 
    except Exception, err:
        p_error = p_error + "get_memory_info().rss: " + str(err) + " : " + 
str(type(err)) + "\n"

    # get num of threads
    try:
        p_num_threads = str(process.get_num_threads())
    except Exception, err:
        p_error = p_error + "get_num_threads(): " + str(err) + " : " + 
str(type(err)) + "\n"

    # get CPU perc 2nd time
    try:
        p_cpu_perc = str(process.get_cpu_percent(interval=0))
    except Exception, err:
        p_error = p_error + "get_cpu_percent() [2nd time]: " + str(err) + " : " 
+ str(type(err)) + "\n"

    # write to XML
    handle.write("<process>\n")
    handle.write("<pid>" + str(process.pid) + "</pid>\n")
    handle.write("<name>" + process.name + "</name>\n")
    handle.write("<cpu_perc>" + p_cpu_perc + "</cpu_perc>\n")
    handle.write("<memory_vms>" + p_mem_vms + "</memory_vms>\n")
    handle.write("<memory_rss>" + p_mem_rss + "</memory_rss>\n")
    handle.write("<num_threads>" + p_num_threads + "</num_threads>\n")

    if len(p_error.strip()) > 0:
        p_error = p_error.replace("&", "&amp;").replace('"', 
"&quot;").replace("<", "&lt;").replace(">", "&gt;").replace("'", 
"&#039;").replace("\n", "<br/>")

    handle.write("<error>" + p_error + "</error>\n")
    handle.write("</process>\n")

handle.write("</processes>")
handle.close()
===

The attached file is the resulting XML when running as administrator. 

If you run the script as normal user, all will have AccessDenied errors. 

On another note, I will submit another ticket on the WinXP crash later. Let's 
keep this as AccessDenied issue.

Attachment: ps.log.xml

[giampaolo]

From [email protected] on May 18, 2011 05:12:09

If looks to me like the only AccessDenied errors are for System processes, 
which are likely legitimate as those are on a different security level than 
Administrator. I don't know that there's anything we can do about that. I don't 
have a Win7 system to confirm - Gimapaolo may be able to help there. 

Non-admin users cannot read process information for processes due to security 
constraints; reading other process' memory space requires SE_DEBUG privileges 
which in turn usually requires an Administrator account. It would be expected 
behavior for things to work as an Administrator user but not as a regular end user acct.

[giampaolo]

From g.rodola on June 04, 2011 05:03:53

I agree with Jay. There's nothing you can do about those AccessDenied 
exceptions.
I'm going to close this out as invalid.

Status: Invalid