Skip to content

docs(benchmarks): verify a quiet box rather than assuming it - #1483

Merged
justinchuby merged 1 commit into
mainfrom
docs/verify-quiet-box
Aug 19, 2026
Merged

justinchuby merged 1 commit into
mainfrom
docs/verify-quiet-box

Conversation

@justinchuby

Copy link
Copy Markdown
Owner

A leaked benchmark process spun a full core on the dev box for eight hours — profile_native pid 9276, 01:00:52 to 09:05, ~5.6 CPU-hours — while casual checks reported the machine free. Every wall-clock figure taken in that window was measured a core short.

Both obvious liveness probes fail, in opposite directions, and the guidance circulating during the session named only the harmless one:

probe failure consequence
nvidia-smi, WMI retains post-mortem rows → false positive looks busy when idle; wastes time
Get-Process -Id <pid> returned absent for that live pid; Stop-Process -Id also failed → false negative looks idle when busy; invites starting a timing run on a contended box

The false negative is the dangerous one, and it is the one we were telling each other to rely on.

What actually worked: a repeated Get-Process -Name <name> snapshot, confirmed by a CPU-time delta over a wall interval — that process gained 5.55 s of CPU in 6 s of wall time. A process whose CPU time advances is alive whatever any single probe says. Terminating it required Get-CimInstance Win32_Process -Filter "ProcessId = <pid>" piped to Invoke-CimMethod -MethodName Terminate.

Also recorded:

  • 0 % GPU utilisation is not evidence of an idle box. A live process holding a CUDA context sits at 0 % between kernels, and a spinning host-side loop consumes a core while showing no GPU activity at all. Two agents independently read 0 % as "free" during this window.
  • Prefer contention-invariant counters where the question allows — CUDA-graph segment counts, htod_bytes, page-in and eviction counts, crash/no-crash ratios, token byte-identity. Several conclusions in this corpus rest on them precisely because a shared box cannot be trusted, and those conclusions are unaffected by this episode.

Sibling to §40 (a threshold calibrated in one regime misfiring in another) and §41 (a shared primitive's contract change breaking distant callers): all three are failure modes that produce confident, wrong results rather than obvious errors.

Docs-only. The measurements quoted are from this box today and the affected throughput figures are being re-taken now that it is genuinely quiet.

A leaked benchmark process spun a full core on the dev box for eight hours
(profile_native pid 9276, 01:00:52 to 09:05, ~5.6 CPU-hours) while casual
checks reported the machine free. Every wall-clock figure taken in that
window was measured a core short.

Both obvious liveness probes fail, in opposite directions, and the guidance
that circulated during the session named only the harmless one:

- nvidia-smi and WMI keep post-mortem rows -> false positive (looks busy
  when idle); wastes time.
- Get-Process -Id returned absent for that live pid, and Stop-Process -Id
  failed against it -> false negative (looks idle when busy); invites
  starting a timing run on a contended box.

What worked was a repeated Get-Process -Name snapshot confirmed by a CPU-time
delta over a wall interval (+5.55 s of CPU in 6 s). Terminating it needed
Get-CimInstance Win32_Process + Invoke-CimMethod Terminate.

Also records that 0% GPU utilisation is not evidence of an idle box, and
that contention-invariant counters should be preferred where the question
allows -- several conclusions in this corpus rest on them for exactly that
reason.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
@justinchuby
justinchuby merged commit 530a460 into main Aug 19, 2026
2 checks passed
@justinchuby
justinchuby deleted the docs/verify-quiet-box branch August 19, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants