Skip to content

Floor memory heartbeat: make the reclaim-throttle wedge attributable from the log - #6502

Merged
briansrls merged 1 commit into
mainfrom
feat/floor-memory-heartbeat
Jul 11, 2026
Merged

briansrls merged 1 commit into
mainfrom
feat/floor-memory-heartbeat

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Lever 2 of the 2026-07-11 memory-mechanism discussion (operator: "lets do 2" — keep it simple and effective). Fidelity-only: reads state, changes no behavior.

Why: the throttle incident's signature — four floors on three hosts with memory.peak pinned at memory.high + <1MiB through 30–49-minute log-silent tails — was invisible in the log because memory.high throttles instead of killing: the kernel swaps the floor to a crawl with zero evidence, and the only receipt was a post-hoc peak read.

What: one synchronous regime-disclosure line at floor start (which memory.high/memory.max bind, and where), then one line per minute from a detached thread: memory.current, memory.swap.current, memory.events high-throttle count, PSI some avg10. The wedge signature becomes current pinned at high, swap climbing, high-events exploding, PSI double-digits — attributable to a 60-second window. Sampling denominator = binding-high dir when set (the slot slice that throttles), else binding-cap, else leaf; absence of all three refuses loudly and the floor proceeds unmonitored — never a fabricated zero.

Shape: binding_cap_cgroup_dir's leaf→root walk factored into tightest_cgroup_dir_for (one walk, two limit files — no copied sibling); parsers pure over file content with discriminating unit tests (exact-key match proven against the oom/oom_kill prefix trap).

Receipts: 3/3 unit tests green; by-execution smoke on the real composed floor (this box, 31.3GiB memory.max regime):

[floor-memory] regime: memory.high=none; memory.max=33578549248 (/sys/fs/cgroup/)
[floor-memory] t=1m current=4265451520 swap=0 high_events=0 psi_some_avg10=0.00
[floor-memory] t=2m current=5804589056 swap=0 high_events=0 psi_some_avg10=0.00

~60 lines/hour worst case against a floor log that prints thousands of SKIP lines.

🤖 Generated with Claude Code

…from the log

Fidelity-only (reads state, changes no behavior). The 2026-07-11 incident
signature - four floors on three hosts with memory.peak pinned at
memory.high + <1MiB through 30-49min log-silent tails - was invisible
because memory.high throttles instead of killing: the kernel swaps the
floor to a crawl with zero log evidence, and the only receipt was a
post-hoc peak read.

One synchronous regime-disclosure line at floor start (which memory.high
and memory.max bind, and where), then one line per minute from a detached
thread: memory.current, memory.swap.current, memory.events high-throttle
count, PSI some avg10. Sampling denominator = the binding-high dir when
set (the slot slice that throttles), else binding-cap, else leaf; absence
of all three refuses loudly and the floor proceeds unmonitored - never a
fabricated zero. binding_cap_cgroup_dir's leaf-to-root walk is factored
into tightest_cgroup_dir_for (one walk, two limit files) instead of a
copied sibling. Parsers are pure over file content with discriminating
unit tests (exact-key match proven against the oom/oom_kill prefix trap).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

The failing ci check is not this diff: batch 3 fails generated_artifact_drift_gate_passes, which is red on main (runs 29165555785 / 29167030699 — Lane D #6474 hand-added cross-links to two generated plan docs without backing .dag rows). This PR's merge-ref inherits it; the fix is #6503. Once #6503 merges, a re-run here goes green — the heartbeat diff itself is Rust-only telemetry with 2/2 approvals. — sent from eager-gull-872

@briansrls
briansrls merged commit 21a8102 into main Jul 11, 2026
2 of 3 checks passed
@briansrls
briansrls deleted the feat/floor-memory-heartbeat branch July 11, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant