Skip to content

fix(kanban): stall veto measures CPU burned now, not lifetime pcpu - #1063

Merged
Kyzcreig merged 1 commit into
mainfrom
fix/kanban-cpu-probe-delta-v2-t_46a2f8a1
Sep 25, 2026
Merged

Kyzcreig merged 1 commit into
mainfrom
fix/kanban-cpu-probe-delta-v2-t_46a2f8a1

Conversation

@Kyzcreig

@Kyzcreig Kyzcreig commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Card: t_46a2f8a1. Supersedes #1028 and #1038 (both conflict with main after #1040).

Bug

_worker_cpu_active read ps -o pcpu. On Linux, procps reports that as lifetime cputime divided by elapsed time. Two effects:

  • Production: a worker that burned CPU once and then stalled read busy for minutes to hours, so it vetoed its own reclaim.
  • Tests: a fresh idle child read busy for about 1 s per 1 ms of startup CPU. That is the _wait_idle flake on loaded runners.

Fix

The probe now takes two psutil cpu_times() samples 0.5 s apart and treats a delta greater than 0 as active. It goes through a new _process_cpu_seconds(pid) sampler seam, which replaces #1040's ps-table seam. Tests inject the sampler.

  • NoSuchProcess (including a zombie) reads idle.
  • Missing psutil, AccessDenied, or any other error reads active, so the probe never authorizes a kill.
  • Only the worker pid is sampled. Children never veto.

Tests

  • Policy tests use an injected _FakeCpu sampler. The production delta logic still makes the decision.
  • Real-probe tests rotate the fault:
    • a fresh in-flight child reads idle at once
    • a once-busy (about 1 s CPU) now-blocked child reads idle at once
    • a genuinely busy child vetoes
    • an exited pid reads idle
  • Probe-failure matrix: AccessDenied, OSError, RuntimeError, psutil missing.

Verification

  • Mac Studio: 15 passed.
  • ACE-AI (Linux, load1 10.8-16.9, 24 cores): 15 passed in each of 3 runs.
  • Mutant return after > 0 (lifetime semantics): 7 of 15 fail on both hosts.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

_worker_cpu_active read `ps -o pcpu`, which on Linux procps is lifetime
cputime/elapsed. A worker that burned CPU once and then stalled read busy
for minutes to hours and vetoed its own reclaim; a fresh idle test child
read busy ~1 s per 1 ms of startup CPU (the t_46a2f8a1 flake).

The probe is now two psutil cpu_times() samples 0.5 s apart (delta > 0 ==
active) through a _process_cpu_seconds sampler seam that replaces #1040's
ps-table seam; tests inject the sampler. NoSuchProcess -> idle; psutil
missing / AccessDenied / any other error -> active (never authorizes a
kill). Children are never sampled.

Real-probe tests rotate the fault: fresh idle child, once-busy-now-blocked
child (~1 s CPU then blocked), genuinely busy child, exited pid.
Supersedes #1028 and #1038.

Verified: test_kanban_progress_stall.py 15 passed (Mac Studio); mutant
`return after > 0` (lifetime semantics) fails 7/15.
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

🤖 merged-by: apollo · lane: discord · gate: BYPASS: FR paused 09-22; gate=CI green + Apollo review · why: t_46a2f8a1: stall-veto probe = psutil cpu_times delta over 0.5s on #1040's injection seam (replaces lifetime ps pcpu: once-busy stalled workers vetoed reclaim forever); fail-safe kept (unknown=>veto, NoSuchProcess=>idle); real-probe tests rotate the fault; supersedes #1028/#1038 (closed); CI 39/39 green on 592aff3; reviewed by Apollo 05:0x PT

@Kyzcreig
Kyzcreig added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit 703b9ce Sep 25, 2026
58 checks passed
@Kyzcreig
Kyzcreig deleted the fix/kanban-cpu-probe-delta-v2-t_46a2f8a1 branch September 25, 2026 15:43
@Kyzcreig Kyzcreig added the fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent) label Sep 25, 2026
@Kyzcreig

Copy link
Copy Markdown
Collaborator Author

FleetReview

Post-merge review (fleetreview:post-merge override): this reviewed the merge commit against its first parent — the bytes that already shipped. It is not a pre-merge gate pass.

Reviewed with 2 of 3 model families — openai unavailable.

Confidence: 4/5

No actionable findings.


FleetReview provenance · models: C=claude-code-opus-5, D=grok-4.6 · cost: $1.84 · duration: 1h 00m 21s · rounds: 3 · files examined: 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fleetreview:post-merge Ask FleetReview to review this MERGED pull (merge commit vs first parent)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant