Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mac Available Memory might be slightly inaccurate #1277

Closed
kgibm opened this issue May 9, 2018 · 10 comments
Closed

Mac Available Memory might be slightly inaccurate #1277

kgibm opened this issue May 9, 2018 · 10 comments

Comments

@kgibm
Copy link

kgibm commented May 9, 2018

psutil's virtual_memory on Mac calculates avail = inactive + free. free comes from psutil_virtual_mem and is calculated as (vm_statistics_data_t.free_count - vm_statistics_data_t.speculative_count) * PAGE_SIZE. This means that avail = (vm_statistics_data_t.inactive_count + vm_statistics_data_t.free_count - vm_statistics_data_t.speculative_count) * PAGE_SIZE. However, I can't find any evidence that speculative_count is effectively unavailable memory. An Apple engineer defines speculative_count as:

on Mac OS X 10.5 we introduced a new, fifth category of memory, speculative memory, used to hold pages that have been read from disk speculatively

The way I interpret this is that the effective amount of free RAM is free - speculative; but, speculative, just by its definition, is probably something that is effectively available for program demands (just like inactive). So I think psutil is calculating free correctly, but avail incorrectly and avail should add back in speculative.

@giampaolo
Copy link
Owner

I'm not a mac user but in general available should report the same value as the system task manager (the GUI one) or "top" cmdline utility at the very least. Perhaps you can check that out?

@kgibm
Copy link
Author

kgibm commented May 14, 2018

Activity Monitor does not have an available statistic:

screenshot

top also does not:

$ top -l 1; vm_stat
Processes: 334 total, 3 running, 331 sleeping, 1296 threads                                                                                                                                                                                                          09:07:10
Load Avg: 3.83, 3.33, 2.71  CPU usage: 17.78% user, 15.22% sys, 66.99% idle  SharedLibs: 213M resident, 58M data, 26M linkedit. MemRegions: 40154 total, 3917M resident, 206M private, 1387M shared. PhysMem: 11G used (2130M wired), 5313M unused.
VM: 1469G vsize, 1114M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 415308/560M in, 119176/11M out. Disks: 1808179/25G read, 285028/5119M written.
[...]
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                                6018.
Pages active:                           1744278.
Pages inactive:                          822029.
Pages speculative:                      1082146.
Pages throttled:                              0.
Pages wired down:                        539463.
Pages purgeable:                           6720.
"Translation faults":                  15928515.
Pages copy-on-write:                     947314.
Pages zero filled:                      7114084.
Pages reactivated:                        15064.
Pages purged:                             44681.
File-backed pages:                      2061989.
Anonymous pages:                        1586464.
Pages stored in compressor:                   0.
Pages occupied by compressor:                 0.
Decompressions:                               0.
Compressions:                                 0.
Pageins:                                3667459.
Pageouts:                                   170.
Swapins:                                      0.
Swapouts:                                     0.

@giampaolo
Copy link
Owner

available mem should be 16 - 7.93.

@kgibm
Copy link
Author

kgibm commented May 21, 2018

I'll close this issue out since I'm also not really a Mac guy and I'm not totally sure of all the details, but for posterity, I'll just assert, that given my reading, and if we want to match the Linux behavior (i.e. subtracting parts of RAM that are available for program demands), then I believe the correct behavior is the equivalent from vm_stat of Pages free + Pages inactive + Pages speculative + File-backed pages.

@kgibm kgibm closed this as completed May 21, 2018
@giampaolo
Copy link
Owner

Please let's keep this open.

@giampaolo giampaolo reopened this May 21, 2018
@giampaolo
Copy link
Owner

OK I think I fixed this in 2f9dcf3. It turns out also used memory was not properly calculated.

@kgibm
Copy link
Author

kgibm commented Oct 16, 2018

@giampaolo That's a very unintuitive Mac API - it seems like the conclusion is that speculative is included in free, but then reported separately, thus why you're not adding speculative to avail and instead subtracting speculative from free.

@giampaolo
Copy link
Owner

giampaolo commented Oct 19, 2018

The point here is that we have no authoritative source establishing how available memory should be calculated. As such I decided to assume two authoritative sources:

  • for available memory we do what zabbix does (avail = inactive + free)
  • for free memory we do what "free" cmdline utility does (free = free - speculative)

I would say the most authoritative source should be the Activity Monitor GUI tool but AFAIK there's no source code for it and it's difficult to guess what it really does under the hoods.

@kgibm
Copy link
Author

kgibm commented Oct 22, 2018

@giampaolo I agree, and thanks for your work on this. I wonder whether speculative should be added to avail, but like you point out maybe that's implicitly in there.

nlevitt added a commit to nlevitt/psutil that referenced this issue Apr 9, 2019
* origin/master: (182 commits)
  giampaolo#1394 / windows / process exe(): convert errno 0 into ERROR_ACCESS_DENIED; errno 0 occurs when the Python process runs in 'Virtual Secure Mode'
  pre-release
  fix win num_handles() test
  update readme
  fix giampaolo#1111: use a lock to make Process.oneshot() thread safe
  pdate HISTORY
  giampaolo#1373: different approach to oneshot() cache (pass Process instances around - which is faster)
  use PROCESS_QUERY_LIMITED_INFORMATION also for username()
  Linux: refactor _parse_stat_file() and return a dict instead of a list (+ maintainability)
  fix giampaolo#1357: do not expose Process' memory_maps() and io_counters() methods if not supported by the kernel
  giampaolo#1376 Windows: check if variable is NULL before free()ing it
  enforce lack of support for Win XP
  fix giampaolo#1370: improper usage of CloseHandle() may lead to override the original error code resulting in raising a wrong exception
  update HISTORY
  (Windows) use PROCESS_QUERY_LIMITED_INFORMATION access rights (giampaolo#1376)
  update HISTORY
  revert 5398c48; let's do it in a separate branch
  giampaolo#1111 make Process.oneshot() thread-safe
  sort HISTORY
  give CREDITS to @EccoTheFlintstone for giampaolo#1368
  fix ionice set not working on windows x64 due to LENGTH_MISMATCH  (giampaolo#1368)
  make flake8 happy
  give CREDITS to @amanusk for giampaolo#1369 / giampaolo#1352 and update doc
  Add CPU frequency support for FreeBSD (giampaolo#1369)
  giampaolo#1359: add test case for cpu_count(logical=False) against lscpu utility
  disable false positive mem test on travis + osx
  fix PEP8 style mistakes
  give credits to @koenkooi for giampaolo#1360
  Fix giampaolo#1354 [Linux] disk_io_counters() fails on Linux kernel 4.18+ (giampaolo#1360)
  giampaolo#1350: give credits to @amanusk
  FreeBSD adding temperature sensors (WIP) (giampaolo#1350)
  pre release
  sensors_temperatures() / linux: convert defaultdict to dict
  fix giampaolo#1004: Process.io_counters() may raise ValueError
  fix giampaolo#1307: [Linux] disk_partitions() does not honour PROCFS_PATH
  refactor hasattr() checks as global constants
  giampaolo#1197 / linux / cpu_freq(): parse /proc/cpuinfo in case /sys/devices/system/cpu fs is not available
  fix giampaolo#1277 / osx / virtual_memory: 'available' and 'used' memory were not calculated properly
  travis / osx: set py 3.6
  travis: disable pypy; se py 3.7 on osx
  skip test on PYPY + Travis
  fix travis
  fix giampaolo#715: do not print exception on import time in case cpu_times() fails.
  fix different travis failures
  give CREDITS for giampaolo#1320 to @truthbk
  [aix] improve compilation on AIX, better support for gcc/g++ + fix cpu metrics (giampaolo#1320)
  give credits to @alxchk for giampaolo#1346 (sunOS)
  Fix giampaolo#1346 (giampaolo#1347)
  giampaolo#1284, giampaolo#1345 - give credits to @amanusk
  Add parsing for /sys/class/thermal (giampaolo#1345)
  Fix decoding error in tests
  catch UnicodeEncodeError on print()
  use memory tolerance in occasionally failing test
  Fix random 0xC0000001 errors when querying for Connections (giampaolo#1335)
  Correct capitalization of PyPI (giampaolo#1337)
  giampaolo#1341: move open() utilities/wrappers in _common.py
  Refactored ps() function in test_posix (giampaolo#1341)
  fix giampaolo#1343: document Process.as_dict() attrs values
  giampaolo#1332 - update HISTORY
  make psutil_debug() aware of PSUTIL_DEBUG (giampaolo#1332)
  also include PYPY (or try to :P)
  travis: add python 3.7 build
  add download badge
  remove failing test assertions
  remove failing test
  make test more robust
  pre release
  pre release
  set version to 5.4.7
  OSX / SMC / sensors: revert giampaolo#1284 (giampaolo#1325)
  setup.py: add py 3.7
  fix giampaolo#1323: [Linux] sensors_temperatures() may fail with ValueError
  fix failing linux tests
  giampaolo#1321 add unit tests
  giampaolo#1321: refactoring
  make disk_io_counters more robust (giampaolo#1324)
  fix typo
  Fix DeprecationWarning: invalid escape sequence (giampaolo#1318)
  remove old test
  update is_storage_device() docstring
  fix giampaolo#1305 / disk_io_counters() / Linux: assume SECTOR_SIZE is a fixed 512
  giampaolo#1313 remove test which no longer makes sense
  disk_io_counters() - linux: mimic iostat behavior (giampaolo#1313)
  fix wrong reference link in doc
  disambiguate TESTFN for parallel testing
  fix giampaolo#1309: add STATUS_PARKED constant and fix STATUS_IDLE (both on linux)
  give CREDITS to @sylvainduchesne for giampaolo#1294
  retain GIL when querying connections table (giampaolo#1306)
  Update index.rst (giampaolo#1308)
  fix giampaolo#1279: catch and skip ENODEV in net_if_stat()
  appveyor: retire 3.5, add 3.7
  revert file renaming of macos files; get them back to 'osx' prefix
  winmake: add upload-wheels cmd
  Rename OSX to macOS (giampaolo#1298)
  apveyor: reset py 3.4 and remove 3.7 (not available yet)
  try to fix occasional children() failure on Win: https://ci.appveyor.com/project/giampaolo/psutil/build/job/je3qyldbb86ff66h
  appveyor: remove py 3.4 and add 3.7
  giampaolo#1284: give credits to @amanusk + some minor adjustments
  little refactoring
  Osx temps (giampaolo#1284)
  ...
@xyzzy-plugh
Copy link

Something seems amiss here; the info from activity monitor doesn't really support the 83.9% coming back from NCPA. What other info can I provide?

macOS 14.4.1, M2 CPU, NCPA 3.0.2

image

https://xxxxxxxxxxxx:5693/api/memory/virtual?units=Gi :
{ "virtual": { "available": [ 2.57, "GiB" ], "total": [ 16, "GiB" ], "percent": [ 83.9, "%" ], "free": [ 1.63, "GiB" ], "used": [ 1.3, "GiB" ] } }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants