Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster Process.children(recursive=True) #1186

Merged
merged 1 commit into from
Dec 2, 2017

Conversation

pitrou
Copy link
Contributor

@pitrou pitrou commented Dec 1, 2017

Before:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
10 loops, best of 3: 29.6 msec per loop

After:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
100 loops, best of 3: 12.4 msec per loop

For reference, non-recursive:

$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children())"
100 loops, best of 3: 12.2 msec per loop

Issue #1183.

Before:
```
$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
10 loops, best of 3: 29.6 msec per loop
```

After:
```
$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children(recursive=True))"
100 loops, best of 3: 12.4 msec per loop
```

For reference, non-recursive:
```
$ python -m timeit -s "import psutil; p = psutil.Process()" "list(p.children())"
100 loops, best of 3: 12.2 msec per loop
```
@pitrou
Copy link
Contributor Author

pitrou commented Dec 1, 2017

@giampaolo

@pitrou
Copy link
Contributor Author

pitrou commented Dec 1, 2017

Basically the idea is that instead of calling Process for all processes on the system and then selecting those that are candidate (grand)children of self, we prune the search space and only call Process for candidate (grand)children.

@pitrou
Copy link
Contributor Author

pitrou commented Dec 1, 2017

(there seems to be an unrelated failure on AppVeyor)

while stack:
pid = stack.pop()
if pid in seen:
# Since pids can be reused while the ppid_map is
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note I'm not sure this check is necessary, as the create_time test may be sufficient to avoid such potential cycles.

@giampaolo
Copy link
Owner

Thanks, this is awesome. I'll try to take a look at this over the week end.

@giampaolo
Copy link
Owner

Looks good. Merging.

@giampaolo giampaolo merged commit f1d94cc into giampaolo:master Dec 2, 2017
@pitrou pitrou deleted the faster_children_recursive branch December 3, 2017 09:53
nlevitt added a commit to nlevitt/psutil that referenced this pull request Apr 9, 2019
* 'pslisten' of github.com:nlevitt/psutil: (922 commits)
  Update INSTALL.rst
  Pass python_requires argument to setuptools (giampaolo#1208)
  giampaolo#1152: fix doc to mention CLI command necessary to enable disk_io_counters() on win
  pre release
  pre release
  pre release
  pre-release
  fix giampaolo#1201: document that timeout kwarg is expressed in seconds
  Add mount points to disk_partitions() in Windows (giampaolo#775) (giampaolo#1192)
  add test for cpu_affinity
  what a stupid bug! (giampaolo#1190)
  update doc
  pre release
  pre-release; also get rid of PSUTIL_DEBUG doc instructions (it's kinda useless for the user after all)
  Use FutureWarning instead of DeprecationWarning (giampaolo#1188)
  fix test
  refactor environ() test
  Fix OSX pid 0 bug (giampaolo#1187)
  change assert in test
  refactor Process.__repr__
  Faster Process.children(recursive=True) (giampaolo#1186)
  Speedup Process.children()  (giampaolo#1185)
  update doc
  update HISTORY
  fix giampaolo#1179 / linux / cmdline: handle processes erroneously overwriting /proc/pid/cmdline by using spaces instead of null bytes as args separator
  set x bit to test_aix.py
  fix giampaolo#1181: raise AD if task_for_pid() fails with 5 and errno == ENOENT
  fix posix failure
  Arguments for NoSuchProcess and AccessDenied for the C ext (giampaolo#1180)
  fix travis failure https://travis-ci.org/giampaolo/psutil/jobs/306424509
  be smarter in searching python exe
  do not test platf specific modules on wheelhouse
  try to fix travis failure
  fix travis failures
  try to use PYTHON_EXE instead of sys.executable
  giampaolo#1177: give credits to @wiggin15
  OSX: implement sensors_battery (giampaolo#1177)
  improve error msg for old windows systems giampaolo#811
  add debug messages
  do not mention apt-get as method of installation as it's not recommended
  syntax highlight in doc files
  syntax highlight in doc files
  fix doc indentation
  1173 debug mode (giampaolo#1176)
  code style
  update MANIFEST
  giampaolo#1174: use TimeoutExpired in wait_pid()
  sort imports by name
  Move exceptions to separate file (giampaolo#1174)
  appveyor: enable python warnings when running tests
  refactor winmake.py
  use a C global variable to figure out whether we're in testing mode
  fix unicode err
  define a setup() function which is called on import by all C modules
  move PyUnicode compt fun definition up in the file
  rename C func
  re-enable test on appveyor; remove unused C code
  refactor PSUTIL_TESTING C APIs
  inspect PSUTIL_TESTING env var from C again
  giampaolo#1152: (DeviceIOControl), skip disk on ERROR_INVALID_FUNCTION and ERROR_NOT_SUPPORTED
  giampaolo#1152 / win / disk_io_counters(): DeviceIOControl errors were ignored; che return value and retry call on ERROR_INSUFFICIENT_BUFFER
  upgrade dist cmds
  change make cmds
  disable IPv6 tests if IPv6 is not supported
  travis / OSX: run py 3.6 instead of 3.4
  fix giampaolo#1169: (Linux) users() hostname returns username instead
  update README, bump up version
  get rid of PSUTIL_TESTING env var: it must be necessarily set from cmdline, hence 'python -m psutil.tests' won't work out of the box
  try to set PSUTIL_TESTING env var from python before failing
  skip cpu_freq tests if not available (giampaolo#1170)
  update doc
  pre-release
  giampaolo#1053: drop python 3.3 support
  try to fix appveyor failure; also refactor generate_manifest.py
  giampaolo#1167 give CREDITS to @matray
  Including non-unicast packets in packet count calculation (giampaolo#1167)
  fix giampaolo#1166 (doc mistake)
  provide a 'make help' command
  ifconfig.py humanize bytes
  try to limit false positives on appveyor/windows
  reap_children() in a finally block in order to limit false positives
  unicode tests: use different name for test dir
  fix failure on osx/travis
  update Makefile
  fix test
  giampaolo#1164 give CREDITS to @wiggin15
  AIX: implement num_ctx_switches (giampaolo#1164)
  use new PYTHON_EXE
  improve logic to determine python exe location
  add DEVNOTES file; move TODO.aix into _psutil_aix.c
  Fix test_emulate_energy_full_not_avail (giampaolo#1163)
  update README
  try to limit occasional appveyor failure
  Remove trove classifiers for untested and unsupported platforms (giampaolo#1162)
  Fix giampaolo#1154: remove 'threads' method on older AIX (giampaolo#1156)
  give CREDITS to @adpag for giampaolo#1159, giampaolo#1160 and giampaolo#1161
  Fix test asserts due to leftover test subprocesses (giampaolo#1161)
  Fix network tests for newer ifconfig versions. (giampaolo#1160)
  Fix pre-commit hook for python 3.x. (giampaolo#1159)
  revert last commit
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants