Relay a cancelled job's SIGTERM to the build process - #4115
Merged
Conversation
…edger actually writes Found live on the first capped job after the cancellation handler shipped (#4108): CIKafka wedged on batch_processing_with_kafka.end_to_end, the stall detector named the test and its pid in the log — and the partial ledger never happened, because the runner signals only the step's own shell and nothing in the bash -> build.sh -> `dotnet run` chain forwards SIGTERM to the grandchild holding the handler. The fix is a pid handshake: registerCancellationCapture publishes Environment.ProcessId to .nuke/temp/build.pid, and the workflow steps run through build/run-with-cancellation-relay.sh, whose INT/TERM trap signals exactly that pid — deliberately NOT the process group, which would also kill the wedged worker the handler wants to dump — and holds the step open while the handler works, inside the runner's own hard-kill deadline. Applied to tests.yml, http.yml, slow-tests.yml and dotnet.yml's test steps; smoke/pack steps have nothing to relay to. Proven by simulating the runner locally: SIGTERM to the relay while CoreTests was mid-run produced "[stall] relaying cancellation to the build process (pid N)" -> "[stall] cancellation signal received" -> a ledger on disk with IsPartial=true, 533 clean passes preserved from the live stream and 2107 honestly Indeterminate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 27, 2026
Open
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap #4108's first real capped job exposed within hours of merging: CIKafka wedged on
batch_processing_with_kafka.end_to_end, the new stall detector named the test and its pid in the log (STALLED: … has been in flight 300s on lane 0 (pid 3814)) — and the partial ledger never happened. The runner signals only the step's own shell on cancellation, and nothing in thebash → build.sh → dotnet runchain forwards SIGTERM to the grandchild whereregisterCancellationCapturelives.The fix: a pid handshake
registerCancellationCapture(build/StallCapture.cs) publishesEnvironment.ProcessIdto.nuke/temp/build.pidwhen it arms.build/run-with-cancellation-relay.sh, whose INT/TERM trap signals exactly that pid — deliberately not the process group, which would also kill the wedged worker the handler wantsdotnet-dumpto capture — then holds the step open (bounded) while the handler writes the ledger and takes stacks, inside the runner's own hard-kill deadline.tests.yml,http.yml,slow-tests.yml, anddotnet.yml's test steps. The smoke/pack steps keep plain./build.sh— no supervisor, nothing to relay to.Proven by simulating the runner locally
SIGTERM sent to the relay while CoreTests was mid-run:
and on disk:
{"IsPartial": true, "CleanPasses": 533, "Indeterminate": 2107}— the 533 verdicts heard on the live stream survive, everything unverdicted is honestly Indeterminate, and the uploaded artifact is what outlives the cancelled job's discarded logs (#4098).Also fixes a compile-time latent in #4111's sibling file:
StallCapture.cswas missingusing Nuke.Common.IO;for theAbsolutePathextensions the pid write needs.🤖 Generated with Claude Code