Replace the CI shell watchdog with Bobcat 0.8.0's supervisor observability - #4108
Merged
Conversation
Bobcat 0.8.0 (JasperFx/bobcat#145-#150) moved everything the CI shell watchdog (build/ci-memory-sampler.sh, GH-4083/GH-4089) approximated from outside the process inside the supervisor, as facts instead of inferences: which TEST is in flight past its budget, in which lane, in which pid; a progress heartbeat; per-test RSS attribution; a pre-kill diagnostic seam; and partial results on cancellation. - StallThreshold 5m (the watchdog's own "a healthy suite is never idle for five minutes" calibration, now per test — no per-target deadline table for heavy suites to keep opting out of), HeartbeatInterval 30s, ResourceSampleInterval 15s. All report-only. - OnBeforeKill runs the watchdog's own dotnet-dump collect + dumpasync --coalesce pipeline (ported to build/StallCapture.cs) against a live wedged worker before the kill discards its state. - SIGTERM/SIGINT under Actions now snapshot the run and write a PARTIAL ledger inside the cancellation grace window (GH-4098: a capped job used to report nothing, and GitHub discards its logs), then best-effort capture the stalled workers' async stacks. - Ledger gains IsPartial / Stalled / StalledTests / PeakWorkerRssMb (all additive; the roll-up jq defaults them for older baselines), with step-summary banners and ::warning annotations; the flakiness roll-up renders partial jobs and stalled tests. - tests.yml sheds the sampler launch and the whole stall_deadline matrix block; build/ci-memory-sampler.sh is deleted. Known trade, stated rather than hidden: the sampler also watched pre-test phases (compile, Docker readiness). The job cap still bounds those, but the automatic dump for that rare case goes away. Smoke: CoreTests 2633/2633 against the published 0.8.0, heartbeats every 30s, ledger fields populated — and the memory attribution already named a real finding (+685 MB retained by one test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 25, 2026
erdtsieck
pushed a commit
to erdtsieck/wolverine
that referenced
this pull request
Aug 27, 2026
…edger actually writes Found live on the first capped job after the cancellation handler shipped (JasperFx#4108): CIKafka wedged on batch_processing_with_kafka.end_to_end, the stall detector named the test and its pid in the log — and the partial ledger never happened, because the runner signals only the step's own shell and nothing in the bash -> build.sh -> `dotnet run` chain forwards SIGTERM to the grandchild holding the handler. The fix is a pid handshake: registerCancellationCapture publishes Environment.ProcessId to .nuke/temp/build.pid, and the workflow steps run through build/run-with-cancellation-relay.sh, whose INT/TERM trap signals exactly that pid — deliberately NOT the process group, which would also kill the wedged worker the handler wants to dump — and holds the step open while the handler works, inside the runner's own hard-kill deadline. Applied to tests.yml, http.yml, slow-tests.yml and dotnet.yml's test steps; smoke/pack steps have nothing to relay to. Proven by simulating the runner locally: SIGTERM to the relay while CoreTests was mid-run produced "[stall] relaying cancellation to the build process (pid N)" -> "[stall] cancellation signal received" -> a ledger on disk with IsPartial=true, 533 clean passes preserved from the live stream and 2107 honestly Indeterminate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4083. Closes #4089. Bobcat 0.8.0 (JasperFx/bobcat#145–#150, cut today) moved everything
build/ci-memory-sampler.sh(#4084/#4090) had to approximate from outside the process inside the supervisor — as facts instead of inferences. The watchdog guessed which pid was the test host (its first version latched ontosqlservr), inferred stalls from flat RSS + idle CPU, and raced the supervisor to the dump. The supervisor now simply states: which test is in flight past its budget, in which lane, in which pid.What changes
build/SupervisedTests.cs— three report-only knobs on every supervised run:StallThreshold = 5m— a test in flight past five minutes is named immediately, in the log and the ledger. Five minutes is the watchdog's own calibration ("a healthy suite is never idle for five minutes"), and per-test in-flight time replaces both its idle heuristic and the per-targetstall_deadlinetable the heavy suites kept having to opt out of — a suite's wall clock no longer matters, only a single test's.HeartbeatInterval = 30s— the one-line progress view (2m32s — 2296/2633 done, 1 in flight (lane 0), longest running: …), so a wedged run's log shows where it stopped and how far it got: the summary a capped job never reaches (CIMarten intermittently wedges and is cancelled at the 20-minute cap — it is not OOM (sampler evidence from #3771) #4083's 18m33s of silence).ResourceSampleInterval = 15s— the sampler's cadence, now with per-test attribution (MartenTests retains ~35MB per test — a GREEN CIMarten run peaks at 9.9GB and ends with 243MB free #4089's 375 MB → 9334 MB growth, finally assignable to tests).build/StallCapture.cs(new) — the two consumer-side pieces Bobcat deliberately doesn't ship:OnBeforeKill: the watchdog'sdotnet-dump collect+dumpasync --coalescepipeline, ported fromasync_stacks(), run against a live wedged worker immediately before the supervisor kills it (dumpasyncis what diagnosed CIPulsar wedges in PulsarNativeReliabilityTests — DotPulsar producers stall in Setup, and the test's 1000s timeout outlives the 20-minute job cap #4100's wedged Pulsar producers). Bounded byBeforeKillTimeout = 10m; a healthy worker never pays for it.Supervisor.Snapshot()→ partial ledger written first (fast — it must land inside the runner's grace window) → stalled tests named with pids → best-effort async-stack capture → exit 2. dotnet.yml's build job wedged and was cancelled at its cap — and the logs were discarded, so there is no evidence #4098's finding was that a capped job reports nothing and GitHub discards its logs; the uploaded artifact is now the thing that survives.build/RetryLedger.cs— additiveIsPartial/Stalled/StalledTests/PeakWorkerRssMbfields, PARTIAL and stalled banners on the step summary, independent::warningannotations.build/flakiness-report.shaggregates them with// 0/// falsedefaults (verified against a mix of new-shape and old-shape ledgers, so an older baseline can't null the roll-up) and renders partial-jobs and stalled-tests sections.tests.yml— the sampler launch,STALL_DEADLINE_SECONDS, and the entirestall_deadlinematrixinclude:block are gone;build/ci-memory-sampler.shis deleted. The measured-ceiling table it carried is obsolete by construction — there is no per-job deadline any more.Known trade, stated rather than hidden
The sampler watched the largest process in the step's tree, which included pre-test phases — compile, Docker readiness gating. Bobcat's stall detection covers tests the supervisor has in flight. A wedge during compile or
DockerUpis still bounded by the job cap, but the automatic dump for that rare case goes away.Verified
CoreTestssupervised against the published 0.8.0: 2633/2633 in 2:52, heartbeats every 30s, ledger fields populated — and the memory attribution immediately produced a real lead:peak worker RSS 5047 MB; top retainer: +685 MB CoreTests.Acceptance.wolverine_as_command_bus.use_iasync_enumerable_as_return_value.bash -nand YAML validation on the workflow.🤖 Generated with Claude Code