Repository navigation
docs(benchmarks): verify a quiet box rather than assuming it - #1483
Merged
Merged
Conversation
A leaked benchmark process spun a full core on the dev box for eight hours (profile_native pid 9276, 01:00:52 to 09:05, ~5.6 CPU-hours) while casual checks reported the machine free. Every wall-clock figure taken in that window was measured a core short. Both obvious liveness probes fail, in opposite directions, and the guidance that circulated during the session named only the harmless one: - nvidia-smi and WMI keep post-mortem rows -> false positive (looks busy when idle); wastes time. - Get-Process -Id returned absent for that live pid, and Stop-Process -Id failed against it -> false negative (looks idle when busy); invites starting a timing run on a contended box. What worked was a repeated Get-Process -Name snapshot confirmed by a CPU-time delta over a wall interval (+5.55 s of CPU in 6 s). Terminating it needed Get-CimInstance Win32_Process + Invoke-CimMethod Terminate. Also records that 0% GPU utilisation is not evidence of an idle box, and that contention-invariant counters should be preferred where the question allows -- several conclusions in this corpus rest on them for exactly that reason. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A leaked benchmark process spun a full core on the dev box for eight hours —
profile_nativepid 9276, 01:00:52 to 09:05, ~5.6 CPU-hours — while casual checks reported the machine free. Every wall-clock figure taken in that window was measured a core short.Both obvious liveness probes fail, in opposite directions, and the guidance circulating during the session named only the harmless one:
nvidia-smi, WMIGet-Process -Id <pid>Stop-Process -Idalso failed → false negativeThe false negative is the dangerous one, and it is the one we were telling each other to rely on.
What actually worked: a repeated
Get-Process -Name <name>snapshot, confirmed by a CPU-time delta over a wall interval — that process gained 5.55 s of CPU in 6 s of wall time. A process whose CPU time advances is alive whatever any single probe says. Terminating it requiredGet-CimInstance Win32_Process -Filter "ProcessId = <pid>"piped toInvoke-CimMethod -MethodName Terminate.Also recorded:
0 %GPU utilisation is not evidence of an idle box. A live process holding a CUDA context sits at 0 % between kernels, and a spinning host-side loop consumes a core while showing no GPU activity at all. Two agents independently read0 %as "free" during this window.htod_bytes, page-in and eviction counts, crash/no-crash ratios, token byte-identity. Several conclusions in this corpus rest on them precisely because a shared box cannot be trusted, and those conclusions are unaffected by this episode.Sibling to §40 (a threshold calibrated in one regime misfiring in another) and §41 (a shared primitive's contract change breaking distant callers): all three are failure modes that produce confident, wrong results rather than obvious errors.
Docs-only. The measurements quoted are from this box today and the affected throughput figures are being re-taken now that it is genuinely quiet.