Skip to content

Relay a cancelled job's SIGTERM to the build process - #4115

Merged
jeremydmiller merged 2 commits into
mainfrom
gh-cancellation-relay
Aug 26, 2026
Merged

Relay a cancelled job's SIGTERM to the build process#4115
jeremydmiller merged 2 commits into
mainfrom
gh-cancellation-relay

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes the gap #4108's first real capped job exposed within hours of merging: CIKafka wedged on batch_processing_with_kafka.end_to_end, the new stall detector named the test and its pid in the log (STALLED: … has been in flight 300s on lane 0 (pid 3814)) — and the partial ledger never happened. The runner signals only the step's own shell on cancellation, and nothing in the bash → build.sh → dotnet run chain forwards SIGTERM to the grandchild where registerCancellationCapture lives.

The fix: a pid handshake

  • registerCancellationCapture (build/StallCapture.cs) publishes Environment.ProcessId to .nuke/temp/build.pid when it arms.
  • Workflow test steps run through build/run-with-cancellation-relay.sh, whose INT/TERM trap signals exactly that pid — deliberately not the process group, which would also kill the wedged worker the handler wants dotnet-dump to capture — then holds the step open (bounded) while the handler writes the ledger and takes stacks, inside the runner's own hard-kill deadline.
  • Applied to tests.yml, http.yml, slow-tests.yml, and dotnet.yml's test steps. The smoke/pack steps keep plain ./build.sh — no supervisor, nothing to relay to.

Proven by simulating the runner locally

SIGTERM sent to the relay while CoreTests was mid-run:

[stall] relaying cancellation to the build process (pid 56144)
[stall] cancellation signal received — writing the partial ledger before the runner discards the run
=== CoreTests: PARTIAL — the run was still in flight when this view was taken
    533 passed, 2107 indeterminate …

and on disk: {"IsPartial": true, "CleanPasses": 533, "Indeterminate": 2107} — the 533 verdicts heard on the live stream survive, everything unverdicted is honestly Indeterminate, and the uploaded artifact is what outlives the cancelled job's discarded logs (#4098).

Also fixes a compile-time latent in #4111's sibling file: StallCapture.cs was missing using Nuke.Common.IO; for the AbsolutePath extensions the pid write needs.

🤖 Generated with Claude Code

jeremydmiller and others added 2 commits August 24, 2026 19:13
…edger actually writes

Found live on the first capped job after the cancellation handler
shipped (#4108): CIKafka wedged on batch_processing_with_kafka.end_to_end,
the stall detector named the test and its pid in the log — and the
partial ledger never happened, because the runner signals only the
step's own shell and nothing in the bash -> build.sh -> `dotnet run`
chain forwards SIGTERM to the grandchild holding the handler.

The fix is a pid handshake: registerCancellationCapture publishes
Environment.ProcessId to .nuke/temp/build.pid, and the workflow steps
run through build/run-with-cancellation-relay.sh, whose INT/TERM trap
signals exactly that pid — deliberately NOT the process group, which
would also kill the wedged worker the handler wants to dump — and
holds the step open while the handler works, inside the runner's own
hard-kill deadline. Applied to tests.yml, http.yml, slow-tests.yml
and dotnet.yml's test steps; smoke/pack steps have nothing to relay to.

Proven by simulating the runner locally: SIGTERM to the relay while
CoreTests was mid-run produced "[stall] relaying cancellation to the
build process (pid N)" -> "[stall] cancellation signal received" ->
a ledger on disk with IsPartial=true, 533 clean passes preserved from
the live stream and 2107 honestly Indeterminate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit c910693 into main Aug 26, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant