Real time disk IO counters

[giampaolo]

From g.rodola on September 15, 2011 00:18:13

Similarly to network_io_counters (see issue 150 ) it would be good for psutil 
to provide a disk_io_counters() function returning the number of bytes read and 
written from/to disk.
With this we'd have the possibility to write tools such as iotop [1].

>>> psutil.disk_io_counters()
iostat(read_bytes=2334342, write_bytes=12353453)

As an extra, if the OS provides the necessary hooks, it might be good to 
provide also the number of read()/write() calls, as per Process.get_io_counters 
method (see issue 64 ).

I haven't looked into any implementation yet so I'm still not sure what the 
implications are and whether this can be done on all platforms.

[1] http://guichaz.free.fr/iotop/

Original issue: http://code.google.com/p/psutil/issues/detail?id=206

[giampaolo]

From [email protected] on September 14, 2011 15:31:43

My request would be to have disk_io_counters() behave just like 
network_io_counters() with a total keyword argument and per-device/partition 
read/write stats.  I'll get cracking on the Mac OS X implementation.

[giampaolo]

From [email protected] on September 20, 2011 12:27:51

From what I see, iostat doesn't distinguish transfer information into 
read/write information.  I'll keep looking but I'm not seeing how this might 
work.  If you know of a better utility to get read/write information for a 
disk, I'll be glad to look at it as a reference.  I also will keep looking into 
what is available to iostat just in case it has the information it needs but 
just doesn't deliver it.

[giampaolo]

From [email protected] on September 20, 2011 12:44:40

It just so happens you can get read/write, I just needed to dig deeper into the 
code.  I'm good to go now.  I'll have the following:

>>> psutil.disk_io_counters()
iostat(name=disk0, reads=1234, writes=1234, read_bytes=1234, write_bytes=1234, 
read_time=1234, write_time=1234)

With all of this, we could reproduce the output of iostat.  What do you think?

[giampaolo]

From g.rodola on September 20, 2011 15:11:45

Good to know you figured this out.
Thinking back about higher level API I think it would be better off to stick 
with something like this (same for psutil.network_io_counters() - issue 150 ; I 
think I'm going to modify that in the same manner):

>>> psutil.disk_io_counters()
iostat(...)
>>> psutil.disk_io_counters(per_disk=True)
{'/dev/sda0': iostat(...),
 '/dev/sda1': iostat(...),
 '/dev/sda2': iostat(...)}
>>>

I think this is better for two reasons:

- total disk io is usually what you want
- a dict is better in case you want to know io stats of a particular partition 
given its mount point (in contrast to iterating over the whole list).

I'm sorry about this, I hope you don't mind.
As for the lower level C API I think it is good to return a list of tuples as such:

[('name', ...), ('name', ...), ('name', ...), ...]

...and then provide the total in python, similarly to what we're currently 
doing here: 
https://code.google.com/p/psutil/source/browse/trunk/psutil/__init__.py#694 
Question: what's read_time/write_time?

[giampaolo]

From [email protected] on September 20, 2011 17:40:28

Do you mind if I commit disk_io_counters prior to your refactoring?  It's not a 
big deal but it would be a lot less work for me.  As for the read/write time, 
it's the total amount of time (in nanoseconds) reading/writing to disk.  It's 
used to create the xfrs/s and MB/s for iostat.  I could see this being very 
useful for tools using psutil for averages/metrics...like I'm doing.  ;)

[giampaolo]

From [email protected] on September 20, 2011 17:41:06

Since I'm asking you to wait, I could have a patch submitted by end of day.  I 
have things working locally and would need to just write a suitable unit test, 
like I did for netstat comparison for network I/O.

[giampaolo]

From [email protected] on September 20, 2011 20:10:12

Here is my current output for OS X:

>>> import psutil
>>> psutil.disk_io_counters()
[iostat(name='disk0', reads=342757L, writes=820590L, read_bytes=5084622336L, 
write_bytes=21940955136L, read_time=1540016653215L, write_time=3884665830052L)]
>>> psutil.disk_io_counters(total=True)
iostat(reads=342758L, writes=820598L, read_bytes=5084626432L, 
write_bytes=21941061632L, read_time=1540029707276L, write_time=3884671096030L)
>>> psutil.disk_io_counters()
[iostat(name='disk2', reads=66L, writes=13L, read_bytes=594944L, 
write_bytes=189952L, read_time=113436291L, write_time=224393426L), 
iostat(name='disk0', reads=342774L, writes=820897L, read_bytes=5084923904L, 
write_bytes=21945732096L, r
ead_time=1540136725056L, write_time=3885044760892L)]
>>> psutil.disk_io_counters(total=True)
iostat(reads=342840L, writes=820910L, read_bytes=5085518848L, 
write_bytes=21945922048L, read_time=1540250161347L, write_time=3885269154318L)

I run it twice, once with only one disk (primary HD) being found and the second 
time after plugging in a USB thumbdrive.  I'll get you a patch when I get a 
unit test up and running.

[giampaolo]

From [email protected] on September 20, 2011 20:15:37

Attached is my preliminary patch.  I followed the psutil.network_io_usage() 
documentation/enablement/testing/approach for this so it should be pretty 
similar.  I did have to start using the CoreFoundation and IOKit frameworks for 
Mac OS X but those are OS supplied so it shouldn't be a big deal.  I also 
skipped on writing a more thorough test since my last one wasn't committed.  We 
can work together to come up with ideal tests if you'd like but for now, it's 
par for the course I'd say.  Let me know what I need to do to get this committed.

Attachment: psutil_issue_206_macosx.diff

[giampaolo]

From g.rodola on September 21, 2011 01:48:48

On OSX 10.5.8 it compiles fine but this is what I get on module import:

users-virtualbox-3:psutil user$ python
Python 2.5.1 ( r251 :54863, Jun 17 2009, 20:37:34) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
/Library/Python/2.5/site-packages/psutil-0.3.0-py2.5-macosx-10.5-i386.egg/_psutil_osx.py:3:
 UserWarning: Module _psutil_osx was already imported from 
/Library/Python/2.5/site-packages/psutil-0.3.0-py2.5-macosx-10.5-i386.egg/_psutil_osx.pyc,
 but /Users/user/psutil is being added to sys.path
  import sys, pkg_resources, imp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "psutil/__init__.py", line 73, in <module>
    import psutil._psosx as _psplatform
  File "psutil/_psosx.py", line 9, in <module>
    import _psutil_osx
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in 
__bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so,
 2): Symbol not found: ___CFConstantStringClassReference
  Referenced from: 
/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so

  Expected in: dynamic lookup

[giampaolo]

From [email protected] on September 21, 2011 05:26:34

Giampaolo, double check and make sure you're not running python with "import 
psutil" while you're in the source dir for psutil. I did that the other day and 
got the same error, because it tries to import psutil from the cwd.

[giampaolo]

From g.rodola on September 21, 2011 05:41:07

I removed the build directory, then re-installed psutil with "python setup.py 
install", then moved into /tmp directory but I still get the same error:

users-virtualbox-3:tmp user$ pwd
/tmp
users-virtualbox-3:tmp user$ python
Python 2.5.1 ( r251 :54863, Jun 17 2009, 20:37:34) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/__init__.py", line 73, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/_psosx.py", line 9, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in 
__bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so,
 2): Symbol not found: ___CFConstantStringClassReference
  Referenced from: 
/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so

  Expected in: dynamic lookup
>>> 

Google brought me to this: https://trac.macports.org/ticket/14758 ...but I'm 
not sure what to do.

[giampaolo]

From [email protected] on September 21, 2011 05:59:42

That gets rid of the UserWarning at least, and for the symbol error for 
___CFConstantStringClassReference it looks like you need to add the flag 
"-framework CoreFoundation" though I don't know offhand what the equivalent 
configuration would be with distutils

[giampaolo]

From [email protected] on September 21, 2011 06:02:25

Answer here: 
http://stackoverflow.com/questions/1676384/how-to-pass-flag-to-gcc-in-python-setup-py-script
 you need to add 

    extra_link_args=['-framework', 'CoreFoundation']

to the Extension definition it looks like.

[giampaolo]

From [email protected] on September 21, 2011 06:05:37

Giampaolo, I committed changes in r1124 to add this flag to setup.py - let me 
know if that works on your 10.5 system.

[giampaolo]

From g.rodola on September 21, 2011 06:30:34

Now error changed into:


>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/__init__.py", line 73, in <module>
  File "build/bdist.macosx-10.5-i386/egg/psutil/_psosx.py", line 9, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 7, in <module>
  File "build/bdist.macosx-10.5-i386/egg/_psutil_osx.py", line 6, in 
__bootstrap__
ImportError: 
dlopen(/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so,
 2): Symbol not found: _kIOMasterPortDefault
  Referenced from: 
/Users/user/.python-eggs/psutil-0.3.0-py2.5-macosx-10.5-i386.egg-tmp/_psutil_osx.so

  Expected in: dynamic lookup

[giampaolo]

From [email protected] on September 21, 2011 06:37:22

Ok, try r1125 which adds -framework IOKit also

[giampaolo]

From g.rodola on September 21, 2011 06:39:52

Ok, it is fixed now, thanks.
I'm going to commit OSX patch soon.

[giampaolo]

From g.rodola on September 21, 2011 06:51:04

Checked in in r1126 .
Later today I'm gonna change the API to use a dict and add the Linux 
implementation.
Just one thing: what do read_time and write_time represent? Seconds spent in 
reading/writing from/to disk?

Status: Started
Labels: Milestone-0.3.1 Progress-1in4

[giampaolo]

From [email protected] on September 21, 2011 06:58:12

Per earlier comment "it's the total amount of time (in nanoseconds) 
reading/writing to disk"

[giampaolo]

From [email protected] on September 21, 2011 09:23:57

You know, I was wondering how it was compiling in the first place.  I had to 
use "-framework CoreFoundation -framework IOKit" when I compiled via gcc but 
when I ran "python setup.py [build|install]" it wasn't complaining and it was 
working.  Weird.  I must have had some left over build or site-package 
installed.  My bad.  (What's funny was I had those extra_compile_args in a 
previous patch and removed them.  *sigh*)

[giampaolo]

From [email protected] on September 21, 2011 09:24:47

s/extra_compile_args/extra_link_args/

[giampaolo]

From [email protected] on September 21, 2011 09:26:43

If it makes you feel any better, it compiled fine here on 10.6 without the 
flags added, so they must be being linked automatically or something on later 
versions of OS X.

[giampaolo]

From [email protected] on September 21, 2011 09:27:10

Crap, I feel so stupid now.  I am still baffled how it worked locally for me in 
the first place.  Anyhow, the answer you found was the solution, using 
extra_link_args in setup.py's Extension for OS X.  Again, I apologize.

[giampaolo]

From [email protected] on September 21, 2011 09:32:09

Yes, that does make me feel much better. ;)

[giampaolo]

From g.rodola on September 21, 2011 11:58:44

API changed as described in comment #4 ( 
https://code.google.com/p/psutil/issues/detail?id=206#c4 ) checked in as r1128 .
I also added some integrity tests. We should add more to compare results 
against iostat cmdline utility.

[giampaolo]

From [email protected] on September 21, 2011 12:01:23

I can submit a patch for comparing to iostat output.  I need to write something 
for work that will do iostat usage/parsing until the next psutil release 
anyways.  I'll submit the patch shortly.

[giampaolo]

From [email protected] on September 21, 2011 13:55:49

For those of you not following Issue 150 , I've taken the suggested refactoring 
for the call/return of disk_io_counters and implemented the same approach for 
network_io_counters.  Should help for consistency.

[giampaolo]

From [email protected] on September 21, 2011 15:37:15

I have created Issue 210 for tracking the enhanced tests required to prove 
psutil.disk_io_counters() is working properly.

[giampaolo]

From g.rodola on September 22, 2011 04:48:45

Linux implementation added in r1130 .

Labels: -Progress-1in4 Progress-2in4

[giampaolo]

From [email protected] on October 13, 2011 09:40:59

Windows implementation added in r1149 .

[giampaolo]

From g.rodola on October 13, 2011 13:13:16

I started looking into BSD implementation.
After reading iostat source code [1] I started using 
devstat_compute_statistics() [2] until I figured out that other that read/write 
bytes it doesn't provide the other 4 information which are useful to us: 
read/write calls and read/write times.

It seems devicestat (9) [3] provides exactly everything we need but I can't 
manage to use it (I'm supposed to use devstat_add_entry() but it's not even 
defined in sys/devicestat.h!).

Anyway, in case it could be useful, I attach this patch based on iostat's 
approach which provides read/write bytes only.

[1] 
http://gitorious.org/freebsd/freebsd/blobs/3bbca39ec7441d70323c6727efa2a865fe3e5442/usr.sbin/iostat/iostat.c#line661
 [2] 
http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=devstat_compute_statistics 
[3] http://www.gsp.com/cgi-bin/man.cgi?section=9&topic=devstat

Attachment: disk-io-bsd.patch

[giampaolo]

From g.rodola on October 21, 2011 15:04:04

Ok, I figured out how to do it.
FreeBSD implementation checked in as r1176 .
This is now implemented on all platforms.

Status: FixedInSVN
Labels: -Progress-2in4 Progress-4in4

[giampaolo]

From g.rodola on October 21, 2011 16:44:17

Labels: -Milestone-0.3.1

[giampaolo]

From g.rodola on October 21, 2011 16:45:27

Labels: Milestone-0.4.0

[giampaolo]

From g.rodola on October 28, 2011 20:44:14

Status: Fixed

[giampaolo]

From g.rodola on March 02, 2013 04:03:32

Updated csets after the SVN -> Mercurial migration: r251 == revision 
24d321225795 r1124 == revision a1d51a7482e4 r1125 == revision b1c1bf0143b1 
r1126 == revision 39750cf12924 r1128 == revision 59b453c08bc9 r1130 == revision 
6ca667aedc68 r1149 == revision f14fbfcf2b6d r1176 == revision edd9514cc641

[talestsp]

I have read that read/write time are "the total amount of time (in nanoseconds) reading/writing to disk" Ok, but when it starts counting?
I have collected these metrics (in a 15 seconds interval) and the values are, for example, 20, 28, 17, 23... I can't understand when this counter is reset to 0.

[giampaolo]

I suppose it starts counting since boot. With newer versions of psutil the counter will never reset to 0.
Please next time do not comment on such an old and closed thread.